Devices, Methods, and Graphical User Interfaces for Generating Tactile Outputs

ABSTRACT

An electronic device receives, at an application-independent module, user interface information from an application. The user interface information corresponds to one or more displayed user interface elements with one or more dimensions defined by an application-specific module of the application. The electronic device receives an input directed toward one or more of the displayed user interface elements, and, at the application-independent module, determines one or more tactile outputs to be generated based on a magnitude of the input and the one or more dimensions defined by the applications-specific module. Using the one or more tactile output generators, the electronic device generates the determined one or more tactile outputs.

RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.16/846,124, filed Apr. 10, 2020, which is a continuation of U.S.application Ser. No. 16/508,218, filed Jul. 10, 2019, now U.S. Pat. No.10,620,708, which is a continuation application of U.S. application Ser.No. 16/240,684, filed Jan. 4, 2019, now U.S. Pat. No. 10,372,221, whichis a continuation application of U.S. application Ser. No. 15/688,754,filed Aug. 28, 2017, now U.S. Pat. No. 10,175,762, which is acontinuation application of U.S. application Ser. No. 15/271,073, filedSep. 20, 2016, now U.S. Pat. No. 9,753,541, which claims priority toU.S. Provisional Application Ser. No. 62/384,159, filed Sep. 6, 2016,all of which are incorporated by reference herein in their entireties.

TECHNICAL FIELD

This relates generally to electronic devices with touch-sensitivesurfaces, including but not limited to electronic devices withtouch-sensitive surfaces that generate tactile outputs to provide hapticfeedback to a user.

BACKGROUND

The use of touch-sensitive surfaces as input devices for computers andother electronic computing devices has increased significantly in recentyears. Example touch-sensitive surfaces include touchpads andtouch-screen displays. Such surfaces are widely used to manipulate userinterfaces and objects therein on a display. Example user interfaceobjects include digital images, video, text, icons, and control elementssuch as buttons and other graphics.

Haptic feedback, typically in combination with visual and/or audiofeedback, is often used in an attempt to make manipulation of userinterfaces and user interface objects more efficient and intuitive for auser, thereby improving the operability of electronic devices.

However, applications that provide different haptic feedback in responseto a same gesture present challenges to users, which may lead tounintended operations. When devices perform unintended operations, theuser needs to cancel such operations and provide inputs again. Thesemanipulations are cumbersome and tedious. In addition, having to undounintended operations and providing inputs again take longer thannecessary, thereby wasting energy. This latter consideration isparticularly important in battery-operated devices. Thus, it would bedesirable to have a framework for providing haptic feedback.

SUMMARY

Accordingly, there is a need for electronic devices with faster, moreefficient, and more consistent methods and interfaces for providinghaptic feedback (typically in conjunction with visual and/or audiofeedback), which in turn make manipulation of user interfaces moreefficient and intuitive for a user. Such methods and interfacesoptionally complement or replace conventional methods for providinghaptic feedback. Such methods and interfaces reduce the number, extent,and/or nature of the inputs from a user by helping the user tounderstand the connection between provided inputs and device responsesto the inputs, thereby creating a more efficient human-machineinterface. In addition, such methods and interfaces provide a moreconsistent human-machine interface for a plurality of softwareapplications. For battery-operated devices, such methods and interfacesconserve power and increase the time between battery charges.

The above deficiencies and other problems associated with userinterfaces for electronic devices with touch-sensitive surfaces arereduced or eliminated by the disclosed devices. In some embodiments, thedevice is a desktop computer. In some embodiments, the device isportable (e.g., a notebook computer, tablet computer, or handhelddevice). In some embodiments, the device is a personal electronic device(e.g., a wearable electronic device, such as a watch). In someembodiments, the device has a touchpad. In some embodiments, the devicehas a touch-sensitive display (also known as a “touch screen” or“touch-screen display”). In some embodiments, the device has a graphicaluser interface (GUI), one or more processors, memory and one or moremodules, programs or sets of instructions stored in the memory forperforming multiple functions. In some embodiments, the user interactswith the GUI primarily through stylus and/or finger contacts andgestures on the touch-sensitive surface. In some embodiments, thefunctions optionally include image editing, drawing, presenting, wordprocessing, spreadsheet making, game playing, telephoning, videoconferencing, e-mailing, instant messaging, workout support, digitalphotographing, digital videoing, web browsing, digital music playing,note taking, and/or digital video playing. Executable instructions forperforming these functions are, optionally, included in a non-transitorycomputer readable storage medium or other computer program productconfigured for execution by one or more processors.

In accordance with some embodiments, a method is performed at anelectronic device with a display, one or more input devices, and one ormore tactile output generators. The method includes displaying, on thedisplay, a user interface for a first software application. The userinterface includes a plurality of elements to which user interactionmodels from a plurality of user interaction models provided by anapplication-independent module have been assigned, the plurality ofelements including a first element to which a first user interactionmodel has been assigned, the plurality of elements having contentprovided by an application-specific module for the first softwareapplication. The first user interaction model defines how the userinterface responds to inputs directed to the first element of the userinterface. The method also includes, while displaying the userinterface, detecting, via the one or more input devices, an inputdirected to the first element of the user interface. The method furtherincludes, in response to detecting the input: updating the userinterface on the display based on characteristics of the input; inaccordance with a determination that the input meets tactile outputcriteria specified by the first user interaction model, generating afirst tactile output corresponding to the input; and, in accordance witha determination that the input does not meet the tactile output criteriaspecified by the first user interaction model, forgoing generation ofthe first tactile output.

In accordance with some embodiments, a method is performed at anelectronic device with a display, a touch-sensitive surface, and one ormore tactile output generators. The method includes displaying, on thedisplay, a user interface; and detecting an occurrence of a firstcondition that triggers a first user interface event that is associatedwith a tactile output. Generating the first user interface eventincludes displaying one or more changes to the user interface. Thedevice will not be able to generate the tactile output associated withthe first user interface event for a respective amount of time. Themethod also includes, in response to detecting the occurrence of thefirst condition, in accordance with a determination that the first userinterface event corresponds to a first user interface event category:delaying generating the first user interface event for at least therespective amount of time; and, after delaying generating the first userinterface event for at least the respective amount of time, displayingthe one or more changes to the user interface and generating the tactileoutput that is associated with the first user interface event. Thedisplay of the one or more changes to the user interface is synchronizedwith the generation of the tactile output that is associated with thefirst user interface event.

In accordance with some embodiments, a method is performed at anelectronic device with a display, a touch-sensitive surface, and one ormore tactile output generators. The method includes, while the one ormore tactile output generators are in a low-power state, receiving anindication that a user interaction has started. A respective portion ofthe user interaction is associated with a tactile output. The low-powerstate is a state in which tactile outputs are generated with a firstamount of latency. The method also includes, in response to receivingthe indication that the user interaction that is associated with thetactile output has started, in accordance with a determination that theindication meets tactile output generator preparation criteria, settingthe one or more tactile output generators to a low-latency state at afirst time. The low-latency state is a state in which the tactileoutputs are generated with a second amount of latency that is lower thanthe first amount of latency. The method further includes, after settingthe one or more tactile output generators to the low-latency state: inaccordance with a determination that the user interaction has reachedthe respective portion of the user interaction that is associated withthe tactile output before a predefined amount of time since the firsttime has elapsed and that the one or more tactile output generators arestill in the low-latency state, generating the tactile output using theone or more tactile output generators with the second amount of latency;and, in accordance with a determination that a tactile output has notbeen generated for at least the predefined amount of time since thefirst time, transitioning the one or more tactile output generators fromthe low-latency state to the low-power state.

In accordance with some embodiments, a method is performed at anelectronic device with a display, a touch-sensitive surface, and one ormore tactile output generators. The method includes, while a first typeof event is associated with a first tactile output and a second type ofevent that is distinct from the first type of event is associated with asecond tactile output that is different from the first tactile output byan application-independent module, receiving first information from afirst application; and, in response to receiving the first informationfrom the first application, generating a response to the firstinformation, including: in accordance with a determination that thefirst information corresponds to an application event of the first type,generating the first tactile output using the one or more tactile outputgenerators; and, in accordance with a determination that the firstinformation corresponds to an application event of the second type thatis distinct from the first type, generating, using the one or moretactile output generators, the second tactile output that is differentfrom the first tactile output. The method also includes, aftergenerating the response to the first information, receiving secondinformation from a second application that is distinct from the firstapplication; and, in response to receiving the second information fromthe second application, generating a response to the second information,including: in accordance with a determination that the secondinformation corresponds to an application event of the first type,generating the first tactile output using the one or more tactile outputgenerators; and, in accordance with a determination that the secondinformation corresponds to an application event of the second type,generating the second tactile output using the one or more tactileoutput generators.

In accordance with some embodiments, a method is performed at anelectronic device with a display, a touch-sensitive surface, and one ormore tactile output generators. The method includes receiving, at anapplication-independent module, from an application-specific module thatis associated with a first application, information about an inputdirected to a user interface of the first application. An operationperformed in the user interface of the first application in response todetecting the input is associated with a tactile output patternspecified by the application-specific module. The information about theinput includes information indicating a magnitude of the operationperformed in the user interface in response to detecting the input. Themethod also includes, in response to receiving the information about theinput directed to the user interface of the first application,generating, via the one or more tactile output generators, a tactileoutput that corresponds to the operation performed in the user interfaceof the first application. The tactile output has the tactile outputpattern specified by the application-specific module. The tactile outputhas an amplitude determined in accordance with the magnitude of theoperation performed in the user interface of the first application inresponse to detecting the input.

In accordance with some embodiments, a method is performed at anelectronic device with a display, a touch-sensitive surface, and one ormore tactile output generators. The method includes displaying a userinterface on the display; and, while displaying the user interface onthe display and while the one or more tactile output generators are in alow-power state, detecting a first user interaction via thetouch-sensitive surface. The method also includes, in response todetecting the first user interaction, setting the one or more tactileoutput generators to a low-latency state; and, after setting the one ormore tactile output generators to the low-latency state, detecting asecond user interaction that is part of a same sequence of userinteractions as the first user interaction. The method further includes,in response to detecting the second user interaction, generating atactile output that corresponds to the second user interaction.

In accordance with some embodiments, a method is performed at anelectronic device with a display, a touch-sensitive surface, and one ormore tactile output generators. The method includes receiving a firstrequest for a first user interface operation. The first user interfaceoperation is a first type of change in a respective user interfaceelement that is associated with a first tactile output. The method alsoincludes, in response to receiving the first request: performing thefirst user interface operation and generating the first tactile output.The method further includes, after performing the first user interfaceoperation, receiving a second request for a second user interfaceoperation that is associated with a second tactile output; and, inresponse to receiving the second request, in accordance with adetermination that the second request for the second user interfaceoperation comprises a request for an instance of the first type ofchange in the user interface that is associated with the second tactileoutput: determining a time interval from a point in time correspondingto a most recent prior instance of the first type of change in therespective user interface element for which a tactile output wasgenerated to a point in time corresponding to the first type of changein the respective user interface element requested by the secondrequest; in accordance with a determination that the time interval isless than a predefined time period, performing the second user interfaceoperation without generating the second tactile output; and, inaccordance with a determination that the time interval is greater thanthe predefined time period, performing the second user interfaceoperation and generating the second tactile output.

In accordance with some embodiments, a method is performed at anelectronic device with a display, a touch-sensitive surface, and one ormore tactile output generators. The method includes, at anapplication-independent module, receiving user interface informationfrom an application. The user interface information corresponds to oneor more displayed user interface elements with one or more dimensionsdefined by an application-specific module of the application. The methodalso includes receiving an input directed toward one or more of thedisplayed user interface elements; and, at the application-independentmodule, determining one or more tactile outputs to be generated based ona magnitude of the input and the one or more dimensions defined by theapplications-specific module; and, using the one or more tactile outputgenerators, generating the determined one or more tactile outputs.

In accordance with some embodiments, an electronic device includes adisplay unit configured to display one or more user interfaces; one ormore input device units configured to receive inputs; one or moretactile output generator units configured to generate one or moretactile outputs; and a processing unit coupled with the display unit,the one or more input device units, and the one or more tactile outputgenerator units. The processing unit is configured to enable display, onthe display unit, of a user interface for a first software application.The user interface includes a plurality of elements to which userinteraction models from a plurality of user interaction models providedby an application-independent module have been assigned, the pluralityof elements including a first element to which a first user interactionmodel has been assigned, the plurality of elements having contentprovided by an application-specific module for the first softwareapplication. The first user interaction model defines how the userinterface responds to inputs directed to the first element of the userinterface. The processing unit is also configured to, while enablingdisplay of the user interface, detect, via the one or more input deviceunits, an input directed to the first element of the user interface;and, in response to detecting the input: update the user interface onthe display unit based on characteristics of the input; in accordancewith a determination that the input meets tactile output criteriaspecified by the first user interaction model, enable generation of afirst tactile output corresponding to the input; and, in accordance witha determination that the input does not meet the tactile output criteriaspecified by the first user interaction model, forgo enabling generationof the first tactile output.

In accordance with some embodiments, an electronic device includes adisplay unit configured to display one or more user interfaces; atouch-sensitive surface unit configured to receive touch inputs; one ormore tactile output generator units configured to generate one or moretactile outputs; and a processing unit coupled with the display unit,the touch-sensitive surface unit, and the one or more tactile outputgenerator units. The processing unit is configured to: enable display,on the display unit, of a user interface; and detect an occurrence of afirst condition that triggers a first user interface event that isassociated with a tactile output. Generating the first user interfaceevent includes enabling display of one or more changes to the userinterface. The device will not be able to generate the tactile outputassociated with the first user interface event for a respective amountof time. The processing unit is also configured to, in response todetecting the occurrence of the first condition, in accordance with adetermination that the first user interface event corresponds to a firstuser interface event category, enable generation of the tactile outputthat is associated with the first user interface event; delay generatingthe first user interface event for at least the respective amount oftime; and, after delaying generating the first user interface event forat least the respective amount of time, enable display of the one ormore changes to the user interface. The display of the one or morechanges to the user interface is synchronized with the generation of thetactile output that is associated with the first user interface event.

In accordance with some embodiments, an electronic device includes adisplay unit configured to display one or more user interfaces; atouch-sensitive surface unit configured to receive touch inputs; one ormore tactile output generator units configured to generate one or moretactile outputs; and a processing unit coupled with the display unit,the touch-sensitive surface unit, and the one or more tactile outputgenerator units. The processing unit is configured to: while the one ormore tactile output generator units are in a low-power state, receive anindication that a user interaction has started. A respective portion ofthe user interaction is associated with a tactile output. The low-powerstate is a state in which tactile outputs are generated with a firstamount of latency. The processing unit is also configured to, inresponse to receiving the indication that the user interaction that isassociated with the tactile output has started, in accordance with adetermination that the indication meets tactile output generatorpreparation criteria, set the one or more tactile output generator unitsto a low-latency state at a first time. The low-latency state is a statein which the tactile outputs are generated with a second amount oflatency that is lower than the first amount of latency. The processingunit is further configured to, after setting the one or more tactileoutput generator units to the low-latency state: in accordance with adetermination that the user interaction has reached the respectiveportion of the user interaction that is associated with the tactileoutput before a predefined amount of time since the first time haselapsed and that the one or more tactile output generator units arestill in the low-latency state, enable generation of the tactile outputusing the one or more tactile output generator units with the secondamount of latency; and, in accordance with a determination that atactile output has not been generated for at least the predefined amountof time since the first time, transition the one or more tactile outputgenerator units from the low-latency state to the low-power state.

In accordance with some embodiments, an electronic device includes adisplay unit configured to display one or more user interfaces; atouch-sensitive surface unit configured to receive touch inputs; one ormore tactile output generator units configured to generate one or moretactile outputs; and a processing unit coupled with the display unit,the one or more input device units, and the one or more tactile outputgenerator units. The processing unit is configured to, while a firsttype of event is associated with a first tactile output and a secondtype of event that is distinct from the first type of event isassociated with a second tactile output that is different from the firsttactile output by an application-independent module: receive firstinformation from a first application; and, in response to receiving thefirst information from the first application, enable generation of aresponse to the first information, including: in accordance with adetermination that the first information corresponds to an applicationevent of the first type, enable generation of the first tactile outputusing the one or more tactile output generator units; and, in accordancewith a determination that the first information corresponds to anapplication event of the second type that is distinct from the firsttype, enable generation, using the one or more tactile output generatorunits, of the second tactile output that is different from the firsttactile output. The processing unit is also configured to, afterenabling generation of the response to the first information, receivesecond information from a second application that is distinct from thefirst application; and, in response to receiving the second informationfrom the second application, enable generation of a response to thesecond information, including: in accordance with a determination thatthe second information corresponds to an application event of the firsttype, enable generation of the first tactile output using the one ormore tactile output generator units; and, in accordance with adetermination that the second information corresponds to an applicationevent of the second type, enable generation of the second tactile outputusing the one or more tactile output generator units.

In accordance with some embodiments, an electronic device includes adisplay unit configured to display one or more user interfaces; atouch-sensitive surface unit configured to receive touch inputs; one ormore tactile output generator units configured to generate one or moretactile outputs; and a processing unit coupled with the display unit,the touch-sensitive surface unit, and the one or more tactile outputgenerator units. The processing unit is configured to receive, at anapplication-independent module, from an application-specific module thatis associated with a first application, information about an inputdirected to a user interface of the first application. An operationperformed in the user interface of the first application in response todetecting the input is associated with a tactile output patternspecified by the application-specific module. The information about theinput includes information indicating a magnitude of the operationperformed in the user interface in response to detecting the input. Theprocessing unit is also configured to, in response to receiving theinformation about the input directed to the user interface of the firstapplication, enable generation, via the one or more tactile outputgenerator units, of a tactile output that corresponds to the operationperformed in the user interface of the first application. The tactileoutput has the tactile output pattern specified by theapplication-specific module. The tactile output has an amplitudedetermined in accordance with the magnitude of the operation performedin the user interface of the first application in response to detectingthe input.

In accordance with some embodiments, an electronic device includes adisplay unit configured to display one or more user interfaces; atouch-sensitive surface unit configured to receive touch inputs; one ormore tactile output generator units configured to generate one or moretactile outputs; and a processing unit coupled with the display unit,the touch-sensitive surface unit, and the one or more tactile outputgenerator units. The processing unit is configured to: enable display ofa user interface on the display unit; while enabling display of the userinterface on the display unit and while the one or more tactile outputgenerator units are in a low-power state, detect a first userinteraction via the touch-sensitive surface unit; in response todetecting the first user interaction, set the one or more tactile outputgenerator units to a low-latency state; after setting the one or moretactile output generator units to the low-latency state, detect a seconduser interaction that is part of a same sequence of user interactions asthe first user interaction; and, in response to detecting the seconduser interaction, enable generation of a tactile output that correspondsto the second user interaction.

In accordance with some embodiments, an electronic device includes adisplay unit configured to display one or more user interfaces; atouch-sensitive surface unit configured to receive touch inputs; one ormore tactile output generator units configured to generate one or moretactile outputs; and a processing unit coupled with the display unit,the touch-sensitive surface unit, and the one or more tactile outputgenerator units. The processing unit is configured to receive a firstrequest for a first user interface operation. The first user interfaceoperation is a first type of change in a respective user interfaceelement that is associated with a first tactile output. The processingunit is also configured to, in response to receiving the first request:perform the first user interface operation and enable generation of thefirst tactile output; and, after performing the first user interfaceoperation, receive a second request for a second user interfaceoperation that is associated with a second tactile output. Theprocessing unit is further configured to, in response to receiving thesecond request, in accordance with a determination that the secondrequest for the second user interface operation comprises a request foran instance of the first type of change in the user interface that isassociated with a second tactile output: determine a time interval froma point in time corresponding to a most recent prior instance of thefirst type of change in the respective user interface element for whicha tactile output was generated to a point in time corresponding to thefirst type of change in the respective user interface element requestedby the second request; in accordance with a determination that the timeinterval is less than a predefined time period, perform the second userinterface operation without enabling generation of the second tactileoutput; and, in accordance with a determination that the time intervalis greater than the predefined time period, perform the second userinterface operation and enable generation of the second tactile output.

In accordance with some embodiments, an electronic device includes adisplay unit configured to display one or more user interfaces; atouch-sensitive surface unit configured to receive touch inputs; one ormore tactile output generator units configured to generate one or moretactile outputs; and a processing unit coupled with the display unit,the touch-sensitive surface unit, and the one or more tactile outputgenerator units. The processing unit is configured to receive, at anapplication-independent module, user interface information from anapplication. The user interface information corresponds to one or moredisplayed user interface elements with one or more dimensions defined byan application-specific module of the application. The processing unitis also configured to receive an input directed toward one or more ofthe displayed user interface elements; determine, at theapplication-independent module, one or more tactile outputs to begenerated based on a magnitude of the input and the one or moredimensions defined by the applications-specific module; and enablegeneration of the determined one or more tactile outputs using the oneor more tactile output generator units.

In accordance with some embodiments, an electronic device includes adisplay, a touch-sensitive surface, optionally one or more sensors todetect intensity of contacts with the touch-sensitive surface, one ormore processors, memory, and one or more programs; the one or moreprograms are stored in the memory and configured to be executed by theone or more processors and the one or more programs include instructionsfor performing or causing performance of the operations of any of themethods described herein. In accordance with some embodiments, acomputer readable storage medium has stored therein instructions whichwhen executed by an electronic device with a display, a touch-sensitivesurface, and optionally one or more sensors to detect intensity ofcontacts with the touch-sensitive surface, cause the device to performor cause performance of the operations of any of the methods describedherein. In accordance with some embodiments, a graphical user interfaceon an electronic device with a display, a touch-sensitive surface,optionally one or more sensors to detect intensity of contacts with thetouch-sensitive surface, a memory, and one or more processors to executeone or more programs stored in the memory includes one or more of theelements displayed in any of the methods described herein, which areupdated in response to inputs, as described in any of the methodsdescribed herein. In accordance with some embodiments, an electronicdevice includes: a display, a touch-sensitive surface, and optionallyone or more sensors to detect intensity of contacts with thetouch-sensitive surface; and means for performing or causing performanceof the operations of any of the methods described herein. In accordancewith some embodiments, an information processing apparatus, for use inan electronic device with a display and a touch-sensitive surface, andoptionally one or more sensors to detect intensity of contacts with thetouch-sensitive surface, includes means for performing or causingperformance of the operations of any of the methods described herein.

Thus, electronic devices with displays, touch-sensitive surfaces andoptionally one or more sensors to detect intensity of contacts with thetouch-sensitive surface are provided with faster, more efficient methodsand interfaces for providing tactile outputs, thereby increasing theeffectiveness, efficiency, and user satisfaction with such devices. Suchmethods and interfaces may complement or replace conventional methodsfor providing tactile outputs.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the various described embodiments,reference should be made to the Description of Embodiments below, inconjunction with the following drawings in which like reference numeralsrefer to corresponding parts throughout the figures.

FIG. 1A is a block diagram illustrating a portable multifunction devicewith a touch-sensitive display in accordance with some embodiments.

FIG. 1B is a block diagram illustrating example components for eventhandling in accordance with some embodiments.

FIG. 1C is a block diagram illustrating a tactile output module inaccordance with some embodiments.

FIGS. 1D-1E are block diagrams illustrating application-independentsoftware modules in accordance with some embodiments.

FIG. 2A illustrates a portable multifunction device having a touchscreen in accordance with some embodiments.

FIGS. 2B-2C show exploded views of an intensity-sensitive input devicein accordance with some embodiments.

FIG. 3 is a block diagram of an example multifunction device with adisplay and a touch-sensitive surface in accordance with someembodiments.

FIG. 4A illustrates an example user interface for a menu of applicationson a portable multifunction device in accordance with some embodiments.

FIG. 4B illustrates an example user interface for a multifunction devicewith a touch-sensitive surface that is separate from the display inaccordance with some embodiments.

FIGS. 4C-4E illustrate examples of dynamic intensity thresholds inaccordance with some embodiments.

FIGS. 4F-4G illustrate a set of sample tactile output patterns inaccordance with some embodiments.

FIGS. 5A-5KKK illustrate example user interfaces and tactile outputs inaccordance with some embodiments.

FIGS. 6A-6D are flow diagrams illustrating a method of generatingtactile outputs based on user interaction models in accordance with someembodiments.

FIG. 7 is a functional block diagram of an electronic device inaccordance with some embodiments.

FIGS. 8A-8D are flow diagrams illustrating a method of synchronizingtactile outputs and user interface changes in accordance with someembodiments.

FIG. 9 is a functional block diagram of an electronic device inaccordance with some embodiments.

FIGS. 10A-10D are flow diagrams illustrating a method of settingpower/latency states of tactile output generators in accordance withsome embodiments.

FIG. 11 is a functional block diagram of an electronic device inaccordance with some embodiments.

FIGS. 12A-12D are flow diagrams illustrating a method of generatingconsistent tactile outputs for multiple applications in accordance withsome embodiments.

FIG. 13 is a functional block diagram of an electronic device inaccordance with some embodiments.

FIGS. 14A-14B are flow diagrams illustrating a method of generatingtactile outputs based on information from an application-specific modulein accordance with some embodiments.

FIG. 15 is a functional block diagram of an electronic device inaccordance with some embodiments.

FIGS. 16A-16D are flow diagrams illustrating a method of settingpower/latency states of tactile output generators based on userinteraction events in accordance with some embodiments.

FIG. 17 is a functional block diagram of an electronic device inaccordance with some embodiments.

FIGS. 18A-18C are flow diagrams illustrating a method of conditionallygenerating tactile outputs based on a time interval from a previoustactile output generation in accordance with some embodiments.

FIG. 19 is a functional block diagram of an electronic device inaccordance with some embodiments.

FIGS. 20A-20B are flow diagrams illustrating a method of generatingtactile outputs based on a magnitude of an input and dimensions of auser interface element in accordance with some embodiments.

FIG. 21 is a functional block diagram of an electronic device inaccordance with some embodiments.

FIG. 22A is a diagram of an example feedback architecture.

FIG. 22B shows an example of a date picker.

FIG. 22C is a diagram showing the overview of an example system.

DESCRIPTION OF EMBODIMENTS

Many electronic devices have graphical user interfaces that allowcertain manipulations of displayed user interface objects in response tocertain touch inputs. However, various software applications may beconfigured to respond in an inconsistent manner (e.g., providingdifferent haptic feedback) to a same input, which makes it morechallenging for a user to learn how to interact with different softwareapplications and/or what particular haptic feedbacks indicate. Thedisclosed embodiments address these limitations and disadvantages byproviding a common framework for providing haptic feedback (and/ortactile outputs). Because instructions for providing tactile outputs areprovided in electronic devices (e.g., in an operating system), softwareapplications can be developed faster and made smaller, thereby improvingthe efficiency in storing and executing such software applications onelectronic devices. In addition, various software applications respondto a same input in a consistent manner (e.g., by providing a consistenthaptic feedback) using the common framework, thereby improving userexperience with such electronic devices. For battery-operated devices,the disclosed methods and devices conserve battery power and increasethe time between battery charges. Furthermore, software applications cansubscribe to different features of the common framework, therebyallowing the software applications to customize responses to inputswithout losing the consistency and advanced features of the commonframework.

Below, FIGS. 1A-1E, 2A-2C, and 3 provide a description of exampledevices. FIGS. 4A-4B and 5A-5KKK illustrate example user interfaces andassociated tactile outputs. FIGS. 6A-6D illustrate a flow diagram of amethod of generating tactile outputs based on user interaction models.FIGS. 8A-8D illustrate a flow diagram of a method of synchronizingtactile outputs and user interface changes. FIGS. 10A-10D illustrate aflow diagram of a method of setting power/latency states of tactileoutput generators. FIGS. 12A-12D illustrate a flow diagram of a methodof generating consistent tactile outputs for multiple applications.FIGS. 14A-14B illustrate a flow diagram of a method of generatingtactile outputs based on information from an application-specificmodule. FIGS. 16A-16D illustrate a flow diagram of a method of settingpower/latency states of tactile output generators based on userinteraction events. FIGS. 18A-18C illustrate a flow diagram of a methodof conditionally generating tactile outputs based on a time intervalfrom a previous tactile output generation. FIGS. 20A-20B illustrate aflow diagram of a method of generating tactile outputs based on amagnitude of an input and dimensions of a user interface element. Theuser interfaces in FIGS. 5A-5KKK are used to illustrate the processes inFIGS. 6A-6D, 8A-8D, 10A-10D, 12A-12D, 14A-14B, 16A-16D, 18A-18C, and20A-20B.

Example Devices

Reference will now be made in detail to embodiments, examples of whichare illustrated in the accompanying drawings. In the following detaileddescription, numerous specific details are set forth in order to providea thorough understanding of the various described embodiments. However,it will be apparent to one of ordinary skill in the art that the variousdescribed embodiments may be practiced without these specific details.In other instances, well-known methods, procedures, components,circuits, and networks have not been described in detail so as not tounnecessarily obscure aspects of the embodiments.

It will also be understood that, although the terms first, second, etc.are, in some instances, used herein to describe various elements, theseelements should not be limited by these terms. These terms are only usedto distinguish one element from another. For example, a first contactcould be termed a second contact, and, similarly, a second contact couldbe termed a first contact, without departing from the scope of thevarious described embodiments. The first contact and the second contactare both contacts, but they are not the same contact, unless the contextclearly indicates otherwise.

The terminology used in the description of the various describedembodiments herein is for the purpose of describing particularembodiments only and is not intended to be limiting. As used in thedescription of the various described embodiments and the appendedclaims, the singular forms “a,” “an,” and “the” are intended to includethe plural forms as well, unless the context clearly indicatesotherwise. It will also be understood that the term “and/or” as usedherein refers to and encompasses any and all possible combinations ofone or more of the associated listed items. It will be furtherunderstood that the terms “includes,” “including,” “comprises,” and/or“comprising,” when used in this specification, specify the presence ofstated features, integers, steps, operations, elements, and/orcomponents, but do not preclude the presence or addition of one or moreother features, integers, steps, operations, elements, components,and/or groups thereof.

As used herein, the term “if” is, optionally, construed to mean “when”or “upon” or “in response to determining” or “in response to detecting,”depending on the context. Similarly, the phrase “if it is determined” or“if [a stated condition or event] is detected” is, optionally, construedto mean “upon determining” or “in response to determining” or “upondetecting [the stated condition or event]” or “in response to detecting[the stated condition or event],” depending on the context.

Embodiments of electronic devices, user interfaces for such devices, andassociated processes for using such devices are described. In someembodiments, the device is a portable communications device, such as amobile telephone, that also contains other functions, such as PDA and/ormusic player functions. Example embodiments of portable multifunctiondevices include, without limitation, the iPhone®, iPod Touch®, and iPad®devices from Apple Inc. of Cupertino, Calif. Other portable electronicdevices, such as laptops or tablet computers with touch-sensitivesurfaces (e.g., touch-screen displays and/or touchpads), are,optionally, used. It should also be understood that, in someembodiments, the device is not a portable communications device, but isa desktop computer with a touch-sensitive surface (e.g., a touch-screendisplay and/or a touchpad).

In the discussion that follows, an electronic device that includes adisplay and a touch-sensitive surface is described. It should beunderstood, however, that the electronic device optionally includes oneor more other physical user-interface devices, such as a physicalkeyboard, a mouse and/or a joystick.

The device typically supports a variety of applications, such as one ormore of the following: a note taking application, a drawing application,a presentation application, a word processing application, a websitecreation application, a disk authoring application, a spreadsheetapplication, a gaming application, a telephone application, a videoconferencing application, an e-mail application, an instant messagingapplication, a workout support application, a photo managementapplication, a digital camera application, a digital video cameraapplication, a web browsing application, a digital music playerapplication, and/or a digital video player application.

The various applications that are executed on the device optionally useat least one common physical user-interface device, such as thetouch-sensitive surface. One or more functions of the touch-sensitivesurface as well as corresponding information displayed on the deviceare, optionally, adjusted and/or varied from one application to the nextand/or within a respective application. In this way, a common physicalarchitecture (such as the touch-sensitive surface) of the deviceoptionally supports the variety of applications with user interfacesthat are intuitive and transparent to the user.

Attention is now directed toward embodiments of portable devices withtouch-sensitive displays. FIG. 1A is a block diagram illustratingportable multifunction device 100 with touch-sensitive display system112 in accordance with some embodiments. Touch-sensitive display system112 is sometimes called a “touch screen” for convenience, and issometimes simply called a touch-sensitive display. Device 100 includesmemory 102 (which optionally includes one or more computer readablestorage mediums), memory controller 122, one or more processing units(CPUs) 120, peripherals interface 118, RF circuitry 108, audio circuitry110, speaker 111, microphone 113, input/output (I/O) subsystem 106,other input or control devices 116, and external port 124. Device 100optionally includes one or more optical sensors 164. Device 100optionally includes one or more intensity sensors 165 for detectingintensity of contacts on device 100 (e.g., a touch-sensitive surfacesuch as touch-sensitive display system 112 of device 100). Device 100optionally includes one or more tactile output generators 167 forgenerating tactile outputs on device 100 (e.g., generating tactileoutputs on a touch-sensitive surface such as touch-sensitive displaysystem 112 of device 100 or touchpad 355 of device 300). Thesecomponents optionally communicate over one or more communication busesor signal lines 103.

As used in the specification and claims, the term “tactile output”refers to physical displacement of a device relative to a previousposition of the device, physical displacement of a component (e.g., atouch-sensitive surface) of a device relative to another component(e.g., housing) of the device, or displacement of the component relativeto a center of mass of the device that will be detected by a user withthe user's sense of touch. For example, in situations where the deviceor the component of the device is in contact with a surface of a userthat is sensitive to touch (e.g., a finger, palm, or other part of auser's hand), the tactile output generated by the physical displacementwill be interpreted by the user as a tactile sensation corresponding toa perceived change in physical characteristics of the device or thecomponent of the device. For example, movement of a touch-sensitivesurface (e.g., a touch-sensitive display or trackpad) is, optionally,interpreted by the user as a “down click” or “up click” of a physicalactuator button. In some cases, a user will feel a tactile sensationsuch as an “down click” or “up click” even when there is no movement ofa physical actuator button associated with the touch-sensitive surfacethat is physically pressed (e.g., displaced) by the user's movements. Asanother example, movement of the touch-sensitive surface is, optionally,interpreted or sensed by the user as “roughness” of the touch-sensitivesurface, even when there is no change in smoothness of thetouch-sensitive surface. While such interpretations of touch by a userwill be subject to the individualized sensory perceptions of the user,there are many sensory perceptions of touch that are common to a largemajority of users. Thus, when a tactile output is described ascorresponding to a particular sensory perception of a user (e.g., an “upclick,” a “down click,” “roughness”), unless otherwise stated, thegenerated tactile output corresponds to physical displacement of thedevice or a component thereof that will generate the described sensoryperception for a typical (or average) user. Using tactile outputs toprovide haptic feedback to a user enhances the operability of the deviceand makes the user-device interface more efficient (e.g., by helping theuser to provide proper inputs and reducing user mistakes whenoperating/interacting with the device) which, additionally, reducespower usage and improves battery life of the device by enabling the userto use the device more quickly and efficiently.

In some embodiments, a tactile output pattern specifies characteristicsof a tactile output, such as the amplitude of the tactile output, theshape of a movement waveform of the tactile output, the frequency of thetactile output, and/or the duration of the tactile output.

When tactile outputs with different tactile output patterns aregenerated by a device (e.g., via one or more tactile output generatorsthat move a movable mass to generate tactile outputs), the tactileoutputs may invoke different haptic sensations in a user holding ortouching the device. While the sensation of the user is based on theuser's perception of the tactile output, most users will be able toidentify changes in waveform, frequency, and amplitude of tactileoutputs generated by the device. Thus, the waveform, frequency andamplitude can be adjusted to indicate to the user that differentoperations have been performed. As such, tactile outputs with tactileoutput patterns that are designed, selected, and/or engineered tosimulate characteristics (e.g., size, material, weight, stiffness,smoothness, etc.); behaviors (e.g., oscillation, displacement,acceleration, rotation, expansion, etc.); and/or interactions (e.g.,collision, adhesion, repulsion, attraction, friction, etc.) of objectsin a given environment (e.g., a user interface that includes graphicalfeatures and objects, a simulated physical environment with virtualboundaries and virtual objects, a real physical environment withphysical boundaries and physical objects, and/or a combination of any ofthe above) will, in some circumstances, provide helpful feedback tousers that reduces input errors and increases the efficiency of theuser's operation of the device. Additionally, tactile outputs are,optionally, generated to correspond to feedback that is unrelated to asimulated physical characteristic, such as an input threshold or aselection of an object. Such tactile outputs will, in somecircumstances, provide helpful feedback to users that reduces inputerrors and increases the efficiency of the user's operation of thedevice.

In some embodiments, a tactile output with a suitable tactile outputpattern serves as a cue for the occurrence of an event of interest in auser interface or behind the scenes in a device. Examples of the eventsof interest include activation of an affordance (e.g., a real or virtualbutton, or toggle switch) provided on the device or in a user interface,success or failure of a requested operation, reaching or crossing aboundary in a user interface, entry into a new state, switching of inputfocus between objects, activation of a new mode, reaching or crossing aninput threshold, detection or recognition of a type of input or gesture,etc. In some embodiments, tactile outputs are provided to serve as awarning or an alert for an impending event or outcome that would occurunless a redirection or interruption input is timely detected. Tactileoutputs are also used in other contexts to enrich the user experience,improve the accessibility of the device to users with visual or motordifficulties or other accessibility needs, and/or improve efficiency andfunctionality of the user interface and/or the device. Tactile outputsare optionally accompanied with audio outputs and/or visible userinterface changes, which further enhance a user's experience when theuser interacts with a user interface and/or the device, and facilitatebetter conveyance of information regarding the state of the userinterface and/or the device, and which reduce input errors and increasethe efficiency of the user's operation of the device.

FIG. 4F provides a set of sample tactile output patterns that may beused, either individually or in combination, either as is or through oneor more transformations (e.g., modulation, amplification, truncation,etc.), to create suitable haptic feedback in various scenarios and forvarious purposes, such as those mentioned above and those described withrespect to the user interfaces and methods discussed herein. Thisexample of a palette of tactile outputs shows how a set of threewaveforms and eight frequencies can be used to produce an array oftactile output patterns. In addition to the tactile output patternsshown in this figure, each of these tactile output patterns isoptionally adjusted in amplitude by changing a gain value for thetactile output pattern, as shown, for example for FullTap 80 Hz, FullTap200 Hz, MiniTap 80 Hz, MiniTap 200 Hz, MicroTap 80 Hz, and MicroTap 200Hz in FIG. 4G, which are each shown with variants having a gain of 1.0,0.75, 0.5, and 0.25. As shown in FIG. 4G, changing the gain of a tactileoutput pattern changes the amplitude of the pattern without changing thefrequency of the pattern or changing the shape of the waveform. In someembodiments, changing the frequency of a tactile output pattern alsoresults in a lower amplitude as some tactile output generators arelimited by how much force can be applied to the movable mass and thushigher frequency movements of the mass are constrained to loweramplitudes to ensure that the acceleration needed to create the waveformdoes not require force outside of an operational force range of thetactile output generator (e.g., the peak amplitudes of the FullTap at230 Hz, 270Hz, and 300 Hz are lower than the amplitudes of the FullTapat 80 Hz, 100 Hz, 125 Hz, and 200 Hz).

In FIG. 4F, each column shows tactile output patterns that have aparticular waveform. The waveform of a tactile output pattern representsthe pattern of physical displacements relative to a neutral position(e.g., x_(zero)) versus time that an movable mass goes through togenerate a tactile output with that tactile output pattern. For example,a first set of tactile output patterns shown in the left column in FIG.4F (e.g., tactile output patterns of a “FullTap”) each have a waveformthat includes an oscillation with two complete cycles (e.g., anoscillation that starts and ends in a neutral position and crosses theneutral position three times). A second set of tactile output patternsshown in the middle column in FIG. 4F (e.g., tactile output patterns ofa “MiniTap”) each have a waveform that includes an oscillation thatincludes one complete cycle (e.g., an oscillation that starts and endsin a neutral position and crosses the neutral position one time). Athird set of tactile output patterns shown in the right column in FIG.4F (e.g., tactile output patterns of a “MicroTap”) each have a waveformthat includes an oscillation that include one half of a complete cycle(e.g., an oscillation that starts and ends in a neutral position anddoes not cross the neutral position). The waveform of a tactile outputpattern also includes a start buffer and an end buffer that representthe gradual speeding up and slowing down of the movable mass at thestart and at the end of the tactile output. The example waveforms shownin FIG. 4F-4G include x_(min) and x_(max) values which represent themaximum and minimum extent of movement of the movable mass. For largerelectronic devices with larger movable masses, there may be larger orsmaller minimum and maximum extents of movement of the mass. The exampleshown in FIGS. 4F-4G describes movement of a mass in 1 dimension,however similar principles would also apply to movement of a movablemass in two or three dimensions.

As shown in FIG. 4F, each tactile output pattern also has acorresponding characteristic frequency that affects the “pitch” of ahaptic sensation that is felt by a user from a tactile output with thatcharacteristic frequency. For a continuous tactile output, thecharacteristic frequency represents the number of cycles that arecompleted within a given period of time (e.g., cycles per second) by themovable mass of the tactile output generator. For a discrete tactileoutput, a discrete output signal (e.g., with 0.5, 1, or 2 cycles) isgenerated, and the characteristic frequency value specifies how fast themovable mass needs to move to generate a tactile output with thatcharacteristic frequency. As shown in FIG. 4F, for each type of tactileoutput (e.g., as defined by a respective waveform, such as FullTap,MiniTap, or MicroTap), a higher frequency value corresponds to fastermovement(s) by the movable mass, and hence, in general, a shorter timeto complete the tactile output (e.g., including the time to complete therequired number of cycle(s) for the discrete tactile output, plus astart and an end buffer time). For example, a FullTap with acharacteristic frequency of 80 Hz takes longer to complete than FullTapwith a characteristic frequency of 100 Hz (e.g., 35.4 ms vs. 28.3 ms inFIG. 4F). In addition, for a given frequency, a tactile output with morecycles in its waveform at a respective frequency takes longer tocomplete than a tactile output with fewer cycles its waveform at thesame respective frequency. For example, a FullTap at 150 Hz takes longerto complete than a MiniTap at 150 Hz (e.g., 19.4 ms vs. 12.8 ms), and aMiniTap at 150 Hz takes longer to complete than a MicroTap at 150 Hz(e.g., 12.8 ms vs. 9.4 ms). However, for tactile output patterns withdifferent frequencies this rule may not apply (e.g., tactile outputswith more cycles but a higher frequency may take a shorter amount oftime to complete than tactile outputs with fewer cycles but a lowerfrequency, and vice versa). For example, at 300 Hz, a FullTap takes aslong as a MiniTap (e.g., 9.9 ms).

As shown in FIG. 4F, a tactile output pattern also has a characteristicamplitude that affects the amount of energy that is contained in atactile signal, or a “strength” of a haptic sensation that may be feltby a user through a tactile output with that characteristic amplitude.In some embodiments, the characteristic amplitude of a tactile outputpattern refers to an absolute or normalized value that represents themaximum displacement of the movable mass from a neutral position whengenerating the tactile output. In some embodiments, the characteristicamplitude of a tactile output pattern is adjustable, e.g., by a fixed ordynamically determined gain factor (e.g., a value between 0 and 1), inaccordance with various conditions (e.g., customized based on userinterface contexts and behaviors) and/or preconfigured metrics (e.g.,input-based metrics, and/or user-interface-based metrics). In someembodiments, an input-based metric (e.g., an intensity-change metric oran input-speed metric) measures a characteristic of an input (e.g., arate of change of a characteristic intensity of a contact in a pressinput or a rate of movement of the contact across a touch-sensitivesurface) during the input that triggers generation of a tactile output.In some embodiments, a user-interface-based metric (e.g., aspeed-across-boundary metric) measures a characteristic of a userinterface element (e.g., a speed of movement of the element across ahidden or visible boundary in a user interface) during the userinterface change that triggers generation of the tactile output. In someembodiments, the characteristic amplitude of a tactile output patternmay be modulated by an “envelope” and the peaks of adjacent cycles mayhave different amplitudes, where one of the waveforms shown above isfurther modified by multiplication by an envelope parameter that changesover time (e.g., from 0 to 1) to gradually adjust amplitude of portionsof the tactile output over time as the tactile output is beinggenerated.

Although specific frequencies, amplitudes, and waveforms are representedin the sample tactile output patterns in FIG. 4F for illustrativepurposes, tactile output patterns with other frequencies, amplitudes,and waveforms may be used for similar purposes. For example, waveformsthat have between 0.5 to 4 cycles can be used. Other frequencies in therange of 60 Hz-400 Hz may be used as well. Table 1 provides examples ofparticular haptic feedback behaviors, configurations, and examples oftheir use.

TABLE 1 Behavior Feedback Configuration Configuration Examples UserInterface Haptics Retarget MicroTap Drag calendar event across dayDefault High (270 Hz) boundary Gain: 0.4 Retarget in force press quickaction Minimum menu Interval: 0.05 Sliding over origin point in ascrubber Reaching 0 degrees when cropping/ straightening Rearranging alist when items snap together Retarget MicroTap Retarget in A-Z scrubberStrong High (270 Hz) Gain: 0.5 Minimum Interval: 0.05 Retarget MicroTapSpinning a wheel in the wheels of Picker High (270 Hz) time userinterface Gain: 0.4 Minimum Interval: 0.05 Impact Default MicroTapChanging scrubbing speed when Medium adjusting a slider (150 Hz)Creating a new calendar event by Gain max: 0.8 tapping and holding Gainmin: 0.0 Activating a toggle switch (changing the switch from on to offor off to on) Reaching a predefined orientation on a compass (e.g.,every 45 degrees from North) Reaching a level state (e.g., 0 degreestilt in any axis for 0.5 seconds) Dropping a pin in a map Sending orreceiving a message with an emphasis animation (e.g., “slam” effect)Sending or receiving an acknowl- edgment of a message Snapping a rulerto different orien- tations (e.g., every 45 degrees) Crossing over asuggested photo while scrubbing through a burst of photos Crossing overa detent in a scrubber (e.g., text size, haptic strength, displaybrightness, display color temperature) Transaction failure notification(ApplePay Failure) Impact Light MicroTap Picking up an existing item(e.g., a Medium calendar event, a favorite in web (150 Hz) browser) Gainmax: 0.6 Moving a time selector over a minor Gain min: 0.0 division oftime (e.g., 15 min) in sleep alarm Impact Strong MicroTap Moving a timeselector over a major Medium division of time (e.g., 1 hour) in (150 Hz)sleep alarm Gain max: 1.0 Gain min: 0.0 Edge Scrubber MicroTap Dragginga brightness scrubber to Medium an edge of the scrubber (150 Hz)Dragging a volume scrubber to an Gain max: 0.6 edge of the scrubber Gainmin: 0.3 Edge Zoom MicroTap Reaching maximum zoom level when High (270Hz) zooming into a photo Gain: 0.6 Re-centering a map Drag DefaultMicroTap Pickup and drop an event in calendar High (270 Hz) Gain Pickup:1.0 Gain Drop: 0.6 Drag Snapping MicroTap Rearrange lists in weather,contacts, High (270 Hz) music, etc. Gain Pickup: 1.0 Gain Drop: 0.6 GainSnap: 1.0 States Swipe Swipe in: Swipe to delete a mail message orAction MiniTap High conversation (270 Hz) Swipe to mark a mail messageas read/ Gain: 1.0 unread in mail Swipe out: Swipe to delete a table row(e.g., MicroTap a document in a document creation/ High (270 Hz) viewingapplication, a note in a Gain: 0.55 notes application, a location in aweather application, a podcast in a podcast application, a song in aplaylist in a music application, a voice memo in a voice recordingapplication Swipe to delete a message while displaying apressure-triggered preview Swipe to mark a message as read/ unread whiledisplaying a pressure-triggered preview Swipe to delete a news articleSwipe to favorite/love a news article Button Default MicroTap Reply tomessage/conversation High (270 Hz) Adding a bookmark in an electronicGain: 0.9 book reader application Activating a virtual assistantStarting to record a voice memo Stopping recording a voice memo ButtonMiniTap Low Delete message/conversation Destructive (100 Hz) FeedbackIntensity: 0.8 Event Success FullTap Confirmation that a payment hasMedium been made (200 Hz) Alert that authentication is needed Gain: 0.7to make a payment (e.g., biometric MiniTap High authentication orpasscode (270 Hz) authentication) Gain: 1.0 Adding a payment account toan electronic wallet application Event Error MiniTap High Failure toprocess a payment transaction (270 Hz) Failure to authenticate afingerprint Gain: 0.85 detected on a fingerprint sensor Gain: 0.75Incorrect passcode/password entered FullTap in a passcode/password entryUI Medium (200 Hz) Gain: 0.65 FullTap Low (150 Hz) Gain: 0.75 EventFullTap High Shake to undo Warning (300 Hz) Gain: 0.9 FullTap Custom(270 Hz) Gain: 0.9 Force Press States Preview MicroTap Peek/Preview(e.g., peek at a mail Custom message) (200 Hz) Gain: 1.0 States PreviewFullTap Pop/Commit (e.g., pop into full mail Custom message) (150 Hz)Gain: 1.0 States Preview MicroTap Unavailable (e.g., press hard on anCustom app icon that doesn't have any (200 Hz) associated quick actions)Gain: 1.0 System Haptics Device MicroTap Press power button once to lockdevice Locked Medium (150 Hz) Gain: 1.0 MiniTap Medium (150 Hz) Gain:1.0 Vibe on Vibe at 150 Hz Attach device to power source Attach thatgradually increases or decreases in amplitude Ringtones & Custom tactileReceive phone call or text message Alerts output using one or more of:Vibe 150 Hz MicroTap 150 Hz MiniTap 150 Hz FullTap 150 Hz Solid-StateHome Button 1 (“Tick”) MiniTap Press home button with click option 230Hz 1 selected Gain: 1.0 2 (“Tak”) MiniTap Press home button with clickoption 270 Hz 2 selected Gain: 1.0 3 (“Tock”) MiniTap Press home buttonwith click option 300 Hz 3 selected Gain: 1.0

The examples shown above in Table 1 are intended to illustrate a rangeof circumstances in which tactile outputs can be generated for differentinputs and events. Table 1 should not be taken as a requirement that adevice respond to each of the listed inputs or events with the indicatedtactile output. Rather, Table 1 is intended to illustrate how tactileoutputs vary and/or are similar for different inputs and/or events(e.g., based on the tactile output pattern, frequency, gain, etc.). Forexample Table 1 shows how an “event success” tactile output varies froman “event failure” tactile output and how a retarget tactile outputdiffers from an impact tactile output.

It should be appreciated that device 100 is only one example of aportable multifunction device, and that device 100 optionally has moreor fewer components than shown, optionally combines two or morecomponents, or optionally has a different configuration or arrangementof the components. The various components shown in FIG. 1A areimplemented in hardware, software, firmware, or a combination thereof,including one or more signal processing and/or application specificintegrated circuits.

Memory 102 optionally includes high-speed random access memory andoptionally also includes non-volatile memory, such as one or moremagnetic disk storage devices, flash memory devices, or othernon-volatile solid-state memory devices. Access to memory 102 by othercomponents of device 100, such as CPU(s) 120 and the peripheralsinterface 118, is, optionally, controlled by memory controller 122.

Peripherals interface 118 can be used to couple input and outputperipherals of the device to CPU(s) 120 and memory 102. The one or moreprocessors 120 run or execute various software programs and/or sets ofinstructions stored in memory 102 to perform various functions fordevice 100 and to process data.

In some embodiments, peripherals interface 118, CPU(s) 120, and memorycontroller 122 are, optionally, implemented on a single chip, such aschip 104. In some other embodiments, they are, optionally, implementedon separate chips.

RF (radio frequency) circuitry 108 receives and sends RF signals, alsocalled electromagnetic signals. RF circuitry 108 converts electricalsignals to/from electromagnetic signals and communicates withcommunications networks and other communications devices via theelectromagnetic signals. RF circuitry 108 optionally includes well-knowncircuitry for performing these functions, including but not limited toan antenna system, an RF transceiver, one or more amplifiers, a tuner,one or more oscillators, a digital signal processor, a CODEC chipset, asubscriber identity module (SIM) card, memory, and so forth. RFcircuitry 108 optionally communicates with networks, such as theInternet, also referred to as the World Wide Web (WWW), an intranetand/or a wireless network, such as a cellular telephone network, awireless local area network (LAN) and/or a metropolitan area network(MAN), and other devices by wireless communication. The wirelesscommunication optionally uses any of a plurality of communicationsstandards, protocols and technologies, including but not limited toGlobal System for Mobile Communications (GSM), Enhanced Data GSMEnvironment (EDGE), high-speed downlink packet access (HSDPA),high-speed uplink packet access (HSDPA), Evolution, Data-Only (EV-DO),HSPA, HSPA+, Dual-Cell HSPA (DC-HSPA), long term evolution (LTE), nearfield communication (NFC), wideband code division multiple access(W-CDMA), code division multiple access (CDMA), time division multipleaccess (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a,IEEE 802.11ac, IEEE 802.11ax, IEEE 802.11b, IEEE 802.11g and/or IEEE802.11n), voice over Internet Protocol (VoIP), Wi-MAX, a protocol fore-mail (e.g., Internet message access protocol (IMAP) and/or post officeprotocol (POP)), instant messaging (e.g., extensible messaging andpresence protocol (XMPP), Session Initiation Protocol for InstantMessaging and Presence Leveraging Extensions (SIMPLE), Instant Messagingand Presence Service (IMPS)), and/or Short Message Service (SMS), or anyother suitable communication protocol, including communication protocolsnot yet developed as of the filing date of this document.

Audio circuitry 110, speaker 111, and microphone 113 provide an audiointerface between a user and device 100. Audio circuitry 110 receivesaudio data from peripherals interface 118, converts the audio data to anelectrical signal, and transmits the electrical signal to speaker 111.Speaker 111 converts the electrical signal to human-audible sound waves.Audio circuitry 110 also receives electrical signals converted bymicrophone 113 from sound waves. Audio circuitry 110 converts theelectrical signal to audio data and transmits the audio data toperipherals interface 118 for processing. Audio data is, optionally,retrieved from and/or transmitted to memory 102 and/or RF circuitry 108by peripherals interface 118. In some embodiments, audio circuitry 110also includes a headset jack (e.g., 212, FIG. 2A). The headset jackprovides an interface between audio circuitry 110 and removable audioinput/output peripherals, such as output-only headphones or a headsetwith both output (e.g., a headphone for one or both ears) and input(e.g., a microphone).

I/O subsystem 106 couples input/output peripherals on device 100, suchas touch-sensitive display system 112 and other input or control devices116, with peripherals interface 118. I/O subsystem 106 optionallyincludes display controller 156, optical sensor controller 158,intensity sensor controller 159, haptic feedback controller 161, and oneor more input controllers 160 for other input or control devices. Theone or more input controllers 160 receive/send electrical signalsfrom/to other input or control devices 116. The other input or controldevices 116 optionally include physical buttons (e.g., push buttons,rocker buttons, etc.), dials, slider switches, joysticks, click wheels,and so forth. In some alternate embodiments, input controller(s) 160are, optionally, coupled with any (or none) of the following: akeyboard, infrared port, USB port, stylus, and/or a pointer device suchas a mouse. The one or more buttons (e.g., 208, FIG. 2A) optionallyinclude an up/down button for volume control of speaker 111 and/ormicrophone 113. The one or more buttons optionally include a push button(e.g., 206, FIG. 2A).

Touch-sensitive display system 112 provides an input interface and anoutput interface between the device and a user. Display controller 156receives and/or sends electrical signals from/to touch-sensitive displaysystem 112. Touch-sensitive display system 112 displays visual output tothe user. The visual output optionally includes graphics, text, icons,video, and any combination thereof (collectively termed “graphics”). Insome embodiments, some or all of the visual output corresponds to userinterface objects. As used herein, the term “affordance” refers to auser-interactive graphical user interface object (e.g., a graphical userinterface object that is configured to respond to inputs directed towardthe graphical user interface object). Examples of user-interactivegraphical user interface objects include, without limitation, a button,slider, icon, selectable menu item, switch, hyperlink, or other userinterface control.

Touch-sensitive display system 112 has a touch-sensitive surface, sensoror set of sensors that accepts input from the user based on hapticand/or tactile contact. Touch-sensitive display system 112 and displaycontroller 156 (along with any associated modules and/or sets ofinstructions in memory 102) detect contact (and any movement or breakingof the contact) on touch-sensitive display system 112 and converts thedetected contact into interaction with user-interface objects (e.g., oneor more soft keys, icons, web pages or images) that are displayed ontouch-sensitive display system 112. In some embodiments, a point ofcontact between touch-sensitive display system 112 and the usercorresponds to a finger of the user or a stylus.

Touch-sensitive display system 112 optionally uses LCD (liquid crystaldisplay) technology, LPD (light emitting polymer display) technology, orLED (light emitting diode) technology, although other displaytechnologies are used in other embodiments. Touch-sensitive displaysystem 112 and display controller 156 optionally detect contact and anymovement or breaking thereof using any of a plurality of touch sensingtechnologies now known or later developed, including but not limited tocapacitive, resistive, infrared, and surface acoustic wave technologies,as well as other proximity sensor arrays or other elements fordetermining one or more points of contact with touch-sensitive displaysystem 112. In some embodiments, projected mutual capacitance sensingtechnology is used, such as that found in the iPhone®, iPod Touch®, andiPad® from Apple Inc. of Cupertino, Calif.

Touch-sensitive display system 112 optionally has a video resolution inexcess of 100 dpi. In some embodiments, the touch screen videoresolution is in excess of 400 dpi (e.g., 500 dpi, 800 dpi, or greater).The user optionally makes contact with touch-sensitive display system112 using any suitable object or appendage, such as a stylus, a finger,and so forth. In some embodiments, the user interface is designed towork with finger-based contacts and gestures, which can be less precisethan stylus-based input due to the larger area of contact of a finger onthe touch screen. In some embodiments, the device translates the roughfinger-based input into a precise pointer/cursor position or command forperforming the actions desired by the user.

In some embodiments, in addition to the touch screen, device 100optionally includes a touchpad (not shown) for activating ordeactivating particular functions. In some embodiments, the touchpad isa touch-sensitive area of the device that, unlike the touch screen, doesnot display visual output. The touchpad is, optionally, atouch-sensitive surface that is separate from touch-sensitive displaysystem 112 or an extension of the touch-sensitive surface formed by thetouch screen.

Device 100 also includes power system 162 for powering the variouscomponents. Power system 162 optionally includes a power managementsystem, one or more power sources (e.g., battery, alternating current(AC)), a recharging system, a power failure detection circuit, a powerconverter or inverter, a power status indicator (e.g., a light-emittingdiode (LED)) and any other components associated with the generation,management and distribution of power in portable devices.

Device 100 optionally also includes one or more optical sensors 164.FIG. 1A shows an optical sensor coupled with optical sensor controller158 in I/O subsystem 106. Optical sensor(s) 164 optionally includecharge-coupled device (CCD) or complementary metal-oxide semiconductor(CMOS) phototransistors. Optical sensor(s) 164 receive light from theenvironment, projected through one or more lens, and converts the lightto data representing an image. In conjunction with imaging module 143(also called a camera module), optical sensor(s) 164 optionally capturestill images and/or video. In some embodiments, an optical sensor islocated on the back of device 100, opposite touch-sensitive displaysystem 112 on the front of the device, so that the touch screen isenabled for use as a viewfinder for still and/or video imageacquisition. In some embodiments, another optical sensor is located onthe front of the device so that the user's image is obtained (e.g., forselfies, for videoconferencing while the user views the other videoconference participants on the touch screen, etc.).

Device 100 optionally also includes one or more contact intensitysensors 165. FIG. 1A shows a contact intensity sensor coupled withintensity sensor controller 159 in I/O subsystem 106. Contact intensitysensor(s) 165 optionally include one or more piezoresistive straingauges, capacitive force sensors, electric force sensors, piezoelectricforce sensors, optical force sensors, capacitive touch-sensitivesurfaces, or other intensity sensors (e.g., sensors used to measure theforce (or pressure) of a contact on a touch-sensitive surface). Contactintensity sensor(s) 165 receive contact intensity information (e.g.,pressure information or a proxy for pressure information) from theenvironment. In some embodiments, at least one contact intensity sensoris collocated with, or proximate to, a touch-sensitive surface (e.g.,touch-sensitive display system 112). In some embodiments, at least onecontact intensity sensor is located on the back of device 100, oppositetouch-screen display system 112 which is located on the front of device100.

Device 100 optionally also includes one or more proximity sensors 166.FIG. 1A shows proximity sensor 166 coupled with peripherals interface118. Alternately, proximity sensor 166 is coupled with input controller160 in I/O subsystem 106. In some embodiments, the proximity sensorturns off and disables touch-sensitive display system 112 when themultifunction device is placed near the user's ear (e.g., when the useris making a phone call).

Device 100 optionally also includes one or more tactile outputgenerators 167. FIG. 1A shows a tactile output generator coupled withhaptic feedback controller 161 in I/O subsystem 106. In someembodiments, tactile output generator(s) 167 optionally include one ormore electroacoustic devices such as speakers or other audio componentsand/or electromechanical devices that convert energy into linear motionsuch as a motor, solenoid, electroactive polymer, piezoelectricactuator, electrostatic actuator, or other tactile output generatingcomponent (e.g., a component that converts electrical signals intotactile outputs on the device). Tactile output generator(s) 167 receivetactile feedback generation instructions from haptic feedback module 133and generates tactile outputs on device 100 that are capable of beingsensed by a user of device 100. In some embodiments, at least onetactile output generator is collocated with, or proximate to, atouch-sensitive surface (e.g., touch-sensitive display system 112) and,optionally, generates a tactile output by moving the touch-sensitivesurface vertically (e.g., in/out of a surface of device 100) orlaterally (e.g., back and forth in the same plane as a surface of device100). In some embodiments, at least one tactile output generator sensoris located on the back of device 100, opposite touch-sensitive displaysystem 112, which is located on the front of device 100.

Device 100 optionally also includes one or more accelerometers 168. FIG.1A shows accelerometer 168 coupled with peripherals interface 118.Alternately, accelerometer 168 is, optionally, coupled with an inputcontroller 160 in I/O subsystem 106. In some embodiments, information isdisplayed on the touch-screen display in a portrait view or a landscapeview based on an analysis of data received from the one or moreaccelerometers. Device 100 optionally includes, in addition toaccelerometer(s) 168, a magnetometer (not shown) and a GPS (or GLONASSor other global navigation system) receiver (not shown) for obtaininginformation concerning the location and orientation (e.g., portrait orlandscape) of device 100.

In some embodiments, the software components stored in memory 102include operating system 126, communication module (or set ofinstructions) 128, contact/motion module (or set of instructions) 130,graphics module (or set of instructions) 132, haptic feedback module (orset of instructions) 133, text input module (or set of instructions)134, Global Positioning System (GPS) module (or set of instructions)135, and applications (or sets of instructions) 136. Furthermore, insome embodiments, memory 102 stores device/global internal state 157, asshown in FIGS. 1A and 3. Device/global internal state 157 includes oneor more of: active application state, indicating which applications, ifany, are currently active; display state, indicating what applications,views or other information occupy various regions of touch-sensitivedisplay system 112; sensor state, including information obtained fromthe device's various sensors and other input or control devices 116; andlocation and/or positional information concerning the device's locationand/or attitude.

Operating system 126 (e.g., iOS, Darwin, RTXC, LINUX, UNIX, OS X,WINDOWS, or an embedded operating system such as VxWorks) includesvarious software components and/or drivers for controlling and managinggeneral system tasks (e.g., memory management, storage device control,power management, etc.) and facilitates communication between varioushardware and software components.

Communication module 128 facilitates communication with other devicesover one or more external ports 124 and also includes various softwarecomponents for handling data received by RF circuitry 108 and/orexternal port 124. External port 124 (e.g., Universal Serial Bus (USB),FIREWIRE, etc.) is adapted for coupling directly to other devices orindirectly over a network (e.g., the Internet, wireless LAN, etc.). Insome embodiments, the external port is a multi-pin (e.g., 30-pin)connector that is the same as, or similar to and/or compatible with the30-pin connector used in some iPhone®, iPod Touch®, and iPad® devicesfrom Apple Inc. of Cupertino, Calif. In some embodiments, the externalport is a Lightning connector that is the same as, or similar to and/orcompatible with the Lightning connector used in some iPhone®, iPodTouch®, and iPad® devices from Apple Inc. of Cupertino, Calif.

Contact/motion module 130 optionally detects contact withtouch-sensitive display system 112 (in conjunction with displaycontroller 156) and other touch-sensitive devices (e.g., a touchpad orphysical click wheel). Contact/motion module 130 includes varioussoftware components for performing various operations related todetection of contact (e.g., by a finger or by a stylus), such asdetermining if contact has occurred (e.g., detecting a finger-downevent), determining an intensity of the contact (e.g., the force orpressure of the contact or a substitute for the force or pressure of thecontact), determining if there is movement of the contact and trackingthe movement across the touch-sensitive surface (e.g., detecting one ormore finger-dragging events), and determining if the contact has ceased(e.g., detecting a finger-up event or a break in contact).Contact/motion module 130 receives contact data from the touch-sensitivesurface. Determining movement of the point of contact, which isrepresented by a series of contact data, optionally includes determiningspeed (magnitude), velocity (magnitude and direction), and/or anacceleration (a change in magnitude and/or direction) of the point ofcontact. These operations are, optionally, applied to single contacts(e.g., one finger contacts or stylus contacts) or to multiplesimultaneous contacts (e.g., “multitouch”/multiple finger contacts). Insome embodiments, contact/motion module 130 and display controller 156detect contact on a touchpad.

Contact/motion module 130 optionally detects a gesture input by a user.Different gestures on the touch-sensitive surface have different contactpatterns (e.g., different motions, timings, and/or intensities ofdetected contacts). Thus, a gesture is, optionally, detected bydetecting a particular contact pattern. For example, detecting a fingertap gesture includes detecting a finger-down event followed by detectinga finger-up (lift off) event at the same position (or substantially thesame position) as the finger-down event (e.g., at the position of anicon). As another example, detecting a finger swipe gesture on thetouch-sensitive surface includes detecting a finger-down event followedby detecting one or more finger-dragging events, and subsequentlyfollowed by detecting a finger-up (lift off) event. Similarly, tap,swipe, drag, and other gestures are optionally detected for a stylus bydetecting a particular contact pattern for the stylus.

In some embodiments, detecting a finger tap gesture (e.g., ontouch-sensitive display system 112) depends on the length of timebetween detecting the finger-down event and the finger-up event, but isindependent of the intensity of the finger contact between detecting thefinger-down event and the finger-up event. In some embodiments, a tapgesture is detected in accordance with a determination that the lengthof time between the finger-down event and the finger-up event is lessthan a predetermined value (e.g., less than 0.1, 0.2, 0.3, 0.4 or 0.5seconds), independent of whether the intensity of the finger contactduring the tap meets a given intensity threshold (greater than a nominalcontact-detection intensity threshold), such as a light press or deeppress intensity threshold. Thus, a finger tap gesture can satisfy inputcriteria that are configured to be met even when the characteristicintensity of a contact does not satisfy a given intensity threshold. Forclarity, the finger contact in a tap gesture typically needs to satisfya nominal contact-detection intensity threshold, below which the contactis not detected, in order for the finger-down event to be detected. Asimilar analysis applies to detecting a tap gesture by a stylus or othercontact. In cases where the device is configured to detect a finger orstylus contact hovering over a touch sensitive surface, the nominalcontact-detection intensity threshold optionally does not correspond tophysical contact between the finger or stylus and the touch sensitivesurface.

The same concepts apply in an analogous manner to other types ofgestures. For example, a swipe gesture, a pinch gesture, a depinchgesture, and/or a long press gesture are optionally detected (e.g., ontouch-sensitive display system 112) based on the satisfaction ofcriteria that are independent of intensities of contacts included in thegesture. For example, a swipe gesture is detected based on an amount ofmovement of one or more contacts; a pinch gesture is detected based onmovement of two or more contacts towards each other; a depinch gestureis detected based on movement of two or more contacts away from eachother; and a long press gesture is detected based on a duration of thecontact on the touch-sensitive surface with less than a threshold amountof movement. As such, the statement that gesture recognition criteriaare configured to be met when a contact in a gesture has an intensitybelow a respective intensity threshold means that the gesturerecognition criteria are capable of being satisfied even if thecontact(s) in the gesture do not reach the respective intensitythreshold. It should be understood, however, that this statement doesnot preclude the gesture recognition criteria from being satisfied incircumstances where one or more of the contacts in the gesture do reachor exceed the respective intensity threshold. For example, a tap gestureis configured to be detected if the finger-down and finger-up event aredetected within a predefined time period, without regard to whether thecontact is above or below the respective intensity threshold during thepredefined time period, and a swipe gesture is configured to be detectedif the contact movement is greater than a predefined magnitude, even ifthe contact is above the respective intensity threshold at the end ofthe contact movement.

Contact intensity thresholds, duration thresholds, and movementthresholds are, in some circumstances, combined in a variety ofdifferent combinations in order to create heuristics for distinguishingtwo or more different gestures directed to the same input element orregion so that multiple different interactions with the same inputelement are enabled to provide a richer set of user interactions andresponses. The statement that a particular set of gesture recognitioncriteria are configured to be met when a contact in a gesture has anintensity below a respective intensity threshold does not preclude theconcurrent evaluation of other intensity-dependent gesture recognitioncriteria to identify other gestures that do have a criteria that is metwhen a gesture includes a contact with an intensity above the respectiveintensity threshold. For example, in some circumstances, first gesturerecognition criteria for a first gesture—which are configured to be metwhen a gesture has an intensity below a respective intensitythreshold—are in competition with second gesture recognition criteriafor a second gesture—which are dependent on the gesture reaching therespective intensity threshold. In such competitions, the gesture is,optionally, not recognized as meeting the first gesture recognitioncriteria for the first gesture if the second gesture recognitioncriteria for the second gesture are met first. For example, if a contactreaches the respective intensity threshold before the contact moves by apredefined amount of movement, a deep press gesture is detected ratherthan a swipe gesture. Conversely, if the contact moves by the predefinedamount of movement before the contact reaches the respective intensitythreshold, a swipe gesture is detected rather than a deep press gesture.Even in such circumstances, the first gesture recognition criteria forthe first gesture are still configured to be met when a contact in thegesture has an intensity below the respective intensity because if thecontact stayed below the respective intensity threshold until an end ofthe gesture (e.g., a swipe gesture with a contact that does not increaseto an intensity above the respective intensity threshold), the gesturewould have been recognized by the first gesture recognition criteria asa swipe gesture. As such, particular gesture recognition criteria thatare configured to be met when an intensity of a contact remains below arespective intensity threshold will (A) in some circumstances ignore theintensity of the contact with respect to the intensity threshold (e.g.for a tap gesture) and/or (B) in some circumstances still be dependenton the intensity of the contact with respect to the intensity thresholdin the sense that the particular gesture recognition criteria (e.g., fora long press gesture) will fail if a competing set ofintensity-dependent gesture recognition criteria (e.g., for a deep pressgesture) recognize an input as corresponding to an intensity-dependentgesture before the particular gesture recognition criteria recognize agesture corresponding to the input (e.g., for a long press gesture thatis competing with a deep press gesture for recognition).

Graphics module 132 includes various known software components forrendering and displaying graphics on touch-sensitive display system 112or other display, including components for changing the visual impact(e.g., brightness, transparency, saturation, contrast or other visualproperty) of graphics that are displayed. As used herein, the term“graphics” includes any object that can be displayed to a user,including without limitation text, web pages, icons (such asuser-interface objects including soft keys), digital images, videos,animations and the like.

In some embodiments, graphics module 132 stores data representinggraphics to be used. Each graphic is, optionally, assigned acorresponding code. Graphics module 132 receives, from applicationsetc., one or more codes specifying graphics to be displayed along with,if necessary, coordinate data and other graphic property data, and thengenerates screen image data to output to display controller 156.

Haptic feedback module 133 includes various software components forgenerating instructions (e.g., instructions used by haptic feedbackcontroller 161) to produce tactile outputs using tactile outputgenerator(s) 167 at one or more locations on device 100 in response touser interactions with device 100.

Text input module 134, which is, optionally, a component of graphicsmodule 132, provides soft keyboards for entering text in variousapplications (e.g., contacts 137, e-mail 140, IM 141, browser 147, andany other application that needs text input).

GPS module 135 determines the location of the device and provides thisinformation for use in various applications (e.g., to telephone 138 foruse in location-based dialing, to camera 143 as picture/video metadata,and to applications that provide location-based services such as weatherwidgets, local yellow page widgets, and map/navigation widgets).

Applications 136 optionally include the following modules (or sets ofinstructions), or a subset or superset thereof:

-   contacts module 137 (sometimes called an address book or contact    list);-   telephone module 138;-   video conferencing module 139;-   e-mail client module 140;-   instant messaging (IM) module 141;-   workout support module 142;-   camera module 143 for still and/or video images;-   image management module 144;-   browser module 147;-   calendar module 148;-   widget modules 149, which optionally include one or more of: weather    widget 149-1, stocks widget 149-2, calculator widget 149-3, alarm    clock widget 149-4, dictionary widget 149-5, and other widgets    obtained by the user, as well as user-created widgets 149-6;-   widget creator module 150 for making user-created widgets 149-6;-   search module 151;-   video and music player module 152, which is, optionally, made up of    a video player module and a music player module;-   notes module 153;-   map module 154; and/or-   online video module 155.

Examples of other applications 136 that are, optionally, stored inmemory 102 include other word processing applications, other imageediting applications, drawing applications, presentation applications,JAVA-enabled applications, encryption, digital rights management, voicerecognition, and voice replication.

In conjunction with touch-sensitive display system 112, displaycontroller 156, contact module 130, graphics module 132, and text inputmodule 134, contacts module 137 includes executable instructions tomanage an address book or contact list (e.g., stored in applicationinternal state 192 of contacts module 137 in memory 102 or memory 370),including: adding name(s) to the address book; deleting name(s) from theaddress book; associating telephone number(s), e-mail address(es),physical address(es) or other information with a name; associating animage with a name; categorizing and sorting names; providing telephonenumbers and/or e-mail addresses to initiate and/or facilitatecommunications by telephone 138, video conference 139, e-mail 140, or IM141; and so forth.

In conjunction with RF circuitry 108, audio circuitry 110, speaker 111,microphone 113, touch-sensitive display system 112, display controller156, contact module 130, graphics module 132, and text input module 134,telephone module 138 includes executable instructions to enter asequence of characters corresponding to a telephone number, access oneor more telephone numbers in address book 137, modify a telephone numberthat has been entered, dial a respective telephone number, conduct aconversation and disconnect or hang up when the conversation iscompleted. As noted above, the wireless communication optionally usesany of a plurality of communications standards, protocols andtechnologies.

In conjunction with RF circuitry 108, audio circuitry 110, speaker 111,microphone 113, touch-sensitive display system 112, display controller156, optical sensor(s) 164, optical sensor controller 158, contactmodule 130, graphics module 132, text input module 134, contact list137, and telephone module 138, videoconferencing module 139 includesexecutable instructions to initiate, conduct, and terminate a videoconference between a user and one or more other participants inaccordance with user instructions.

In conjunction with RF circuitry 108, touch-sensitive display system112, display controller 156, contact module 130, graphics module 132,and text input module 134, e-mail client module 140 includes executableinstructions to create, send, receive, and manage e-mail in response touser instructions. In conjunction with image management module 144,e-mail client module 140 makes it very easy to create and send e-mailswith still or video images taken with camera module 143.

In conjunction with RF circuitry 108, touch-sensitive display system112, display controller 156, contact module 130, graphics module 132,and text input module 134, the instant messaging module 141 includesexecutable instructions to enter a sequence of characters correspondingto an instant message, to modify previously entered characters, totransmit a respective instant message (for example, using a ShortMessage Service (SMS) or Multimedia Message Service (MMS) protocol fortelephony-based instant messages or using XMPP, SIMPLE, Apple PushNotification Service (APNs) or IMPS for Internet-based instantmessages), to receive instant messages and to view received instantmessages. In some embodiments, transmitted and/or received instantmessages optionally include graphics, photos, audio files, video filesand/or other attachments as are supported in a MMS and/or an EnhancedMessaging Service (EMS). As used herein, “instant messaging” refers toboth telephony-based messages (e.g., messages sent using SMS or MMS) andInternet-based messages (e.g., messages sent using XMPP, SIMPLE, APNs,or IMPS).

In conjunction with RF circuitry 108, touch-sensitive display system112, display controller 156, contact module 130, graphics module 132,text input module 134, GPS module 135, map module 154, and music playermodule 146, workout support module 142 includes executable instructionsto create workouts (e.g., with time, distance, and/or calorie burninggoals); communicate with workout sensors (in sports devices and smartwatches); receive workout sensor data; calibrate sensors used to monitora workout; select and play music for a workout; and display, store andtransmit workout data.

In conjunction with touch-sensitive display system 112, displaycontroller 156, optical sensor(s) 164, optical sensor controller 158,contact module 130, graphics module 132, and image management module144, camera module 143 includes executable instructions to capture stillimages or video (including a video stream) and store them into memory102, modify characteristics of a still image or video, and/or delete astill image or video from memory 102.

In conjunction with touch-sensitive display system 112, displaycontroller 156, contact module 130, graphics module 132, text inputmodule 134, and camera module 143, image management module 144 includesexecutable instructions to arrange, modify (e.g., edit), or otherwisemanipulate, label, delete, present (e.g., in a digital slide show oralbum), and store still and/or video images.

In conjunction with RF circuitry 108, touch-sensitive display system112, display system controller 156, contact module 130, graphics module132, and text input module 134, browser module 147 includes executableinstructions to browse the Internet in accordance with userinstructions, including searching, linking to, receiving, and displayingweb pages or portions thereof, as well as attachments and other fileslinked to web pages.

In conjunction with RF circuitry 108, touch-sensitive display system112, display system controller 156, contact module 130, graphics module132, text input module 134, e-mail client module 140, and browser module147, calendar module 148 includes executable instructions to create,display, modify, and store calendars and data associated with calendars(e.g., calendar entries, to do lists, etc.) in accordance with userinstructions.

In conjunction with RF circuitry 108, touch-sensitive display system112, display system controller 156, contact module 130, graphics module132, text input module 134, and browser module 147, widget modules 149are mini-applications that are, optionally, downloaded and used by auser (e.g., weather widget 149-1, stocks widget 149-2, calculator widget149-3, alarm clock widget 149-4, and dictionary widget 149-5) or createdby the user (e.g., user-created widget 149-6). In some embodiments, awidget includes an HTML (Hypertext Markup Language) file, a CSS(Cascading Style Sheets) file, and a JavaScript file. In someembodiments, a widget includes an XML (Extensible Markup Language) fileand a JavaScript file (e.g., Yahoo! Widgets).

In conjunction with RF circuitry 108, touch-sensitive display system112, display system controller 156, contact module 130, graphics module132, text input module 134, and browser module 147, the widget creatormodule 150 includes executable instructions to create widgets (e.g.,turning a user-specified portion of a web page into a widget).

In conjunction with touch-sensitive display system 112, display systemcontroller 156, contact module 130, graphics module 132, and text inputmodule 134, search module 151 includes executable instructions to searchfor text, music, sound, image, video, and/or other files in memory 102that match one or more search criteria (e.g., one or more user-specifiedsearch terms) in accordance with user instructions.

In conjunction with touch-sensitive display system 112, display systemcontroller 156, contact module 130, graphics module 132, audio circuitry110, speaker 111, RF circuitry 108, and browser module 147, video andmusic player module 152 includes executable instructions that allow theuser to download and play back recorded music and other sound filesstored in one or more file formats, such as MP3 or AAC files, andexecutable instructions to display, present or otherwise play backvideos (e.g., on touch-sensitive display system 112, or on an externaldisplay connected wirelessly or via external port 124). In someembodiments, device 100 optionally includes the functionality of an MP3player, such as an iPod (trademark of Apple Inc.).

In conjunction with touch-sensitive display system 112, displaycontroller 156, contact module 130, graphics module 132, and text inputmodule 134, notes module 153 includes executable instructions to createand manage notes, to do lists, and the like in accordance with userinstructions.

In conjunction with RF circuitry 108, touch-sensitive display system112, display system controller 156, contact module 130, graphics module132, text input module 134, GPS module 135, and browser module 147, mapmodule 154 includes executable instructions to receive, display, modify,and store maps and data associated with maps (e.g., driving directions;data on stores and other points of interest at or near a particularlocation; and other location-based data) in accordance with userinstructions.

In conjunction with touch-sensitive display system 112, display systemcontroller 156, contact module 130, graphics module 132, audio circuitry110, speaker 111, RF circuitry 108, text input module 134, e-mail clientmodule 140, and browser module 147, online video module 155 includesexecutable instructions that allow the user to access, browse, receive(e.g., by streaming and/or download), play back (e.g., on the touchscreen 112, or on an external display connected wirelessly or viaexternal port 124), send an e-mail with a link to a particular onlinevideo, and otherwise manage online videos in one or more file formats,such as H.264. In some embodiments, instant messaging module 141, ratherthan e-mail client module 140, is used to send a link to a particularonline video.

Each of the above identified modules and applications correspond to aset of executable instructions for performing one or more functionsdescribed above and the methods described in this application (e.g., thecomputer-implemented methods and other information processing methodsdescribed herein). These modules (i.e., sets of instructions) need notbe implemented as separate software programs, procedures or modules, andthus various subsets of these modules are, optionally, combined orotherwise re-arranged in various embodiments. In some embodiments,memory 102 optionally stores a subset of the modules and data structuresidentified above. Furthermore, memory 102 optionally stores additionalmodules and data structures not described above.

In some embodiments, device 100 is a device where operation of apredefined set of functions on the device is performed exclusivelythrough a touch screen and/or a touchpad. By using a touch screen and/ora touchpad as the primary input control device for operation of device100, the number of physical input control devices (such as push buttons,dials, and the like) on device 100 is, optionally, reduced.

The predefined set of functions that are performed exclusively through atouch screen and/or a touchpad optionally include navigation betweenuser interfaces. In some embodiments, the touchpad, when touched by theuser, navigates device 100 to a main, home, or root menu from any userinterface that is displayed on device 100. In such embodiments, a “menubutton” is implemented using a touchpad. In some other embodiments, themenu button is a physical push button or other physical input controldevice instead of a touchpad.

FIG. 1B is a block diagram illustrating example components for eventhandling in accordance with some embodiments. In some embodiments,memory 102 (in FIG. 1A) or 370 (FIG. 3) includes event sorter 170 (e.g.,in operating system 126) and a respective application 136-1 (e.g., anyof the aforementioned applications 136, 137-155, 380-390).

Event sorter 170 receives event information and determines theapplication 136-1 and application view 191 of application 136-1 to whichto deliver the event information. Event sorter 170 includes eventmonitor 171 and event dispatcher module 174. In some embodiments,application 136-1 includes application internal state 192, whichindicates the current application view(s) displayed on touch-sensitivedisplay system 112 when the application is active or executing. In someembodiments, device/global internal state 157 is used by event sorter170 to determine which application(s) is (are) currently active, andapplication internal state 192 is used by event sorter 170 to determineapplication views 191 to which to deliver event information.

In some embodiments, application internal state 192 includes additionalinformation, such as one or more of: resume information to be used whenapplication 136-1 resumes execution, user interface state informationthat indicates information being displayed or that is ready for displayby application 136-1, a state queue for enabling the user to go back toa prior state or view of application 136-1, and a redo/undo queue ofprevious actions taken by the user.

Event monitor 171 receives event information from peripherals interface118. Event information includes information about a sub-event (e.g., auser touch on touch-sensitive display system 112, as part of amulti-touch gesture). Peripherals interface 118 transmits information itreceives from I/O subsystem 106 or a sensor, such as proximity sensor166, accelerometer(s) 168, and/or microphone 113 (through audiocircuitry 110). Information that peripherals interface 118 receives fromI/O subsystem 106 includes information from touch-sensitive displaysystem 112 or a touch-sensitive surface.

In some embodiments, event monitor 171 sends requests to the peripheralsinterface 118 at predetermined intervals. In response, peripheralsinterface 118 transmits event information. In other embodiments,peripheral interface 118 transmits event information only when there isa significant event (e.g., receiving an input above a predeterminednoise threshold and/or for more than a predetermined duration).

In some embodiments, event sorter 170 also includes a hit viewdetermination module 172 and/or an active event recognizer determinationmodule 173.

Hit view determination module 172 provides software procedures fordetermining where a sub-event has taken place within one or more views,when touch-sensitive display system 112 displays more than one view.Views are made up of controls and other elements that a user can see onthe display.

Another aspect of the user interface associated with an application is aset of views, sometimes herein called application views or userinterface windows, in which information is displayed and touch-basedgestures occur. The application views (of a respective application) inwhich a touch is detected optionally correspond to programmatic levelswithin a programmatic or view hierarchy of the application. For example,the lowest level view in which a touch is detected is, optionally,called the hit view, and the set of events that are recognized as properinputs are, optionally, determined based, at least in part, on the hitview of the initial touch that begins a touch-based gesture.

Hit view determination module 172 receives information related tosub-events of a touch-based gesture. When an application has multipleviews organized in a hierarchy, hit view determination module 172identifies a hit view as the lowest view in the hierarchy which shouldhandle the sub-event. In most circumstances, the hit view is the lowestlevel view in which an initiating sub-event occurs (i.e., the firstsub-event in the sequence of sub-events that form an event or potentialevent). Once the hit view is identified by the hit view determinationmodule, the hit view typically receives all sub-events related to thesame touch or input source for which it was identified as the hit view.

Active event recognizer determination module 173 determines which viewor views within a view hierarchy should receive a particular sequence ofsub-events. In some embodiments, active event recognizer determinationmodule 173 determines that only the hit view should receive a particularsequence of sub-events. In other embodiments, active event recognizerdetermination module 173 determines that all views that include thephysical location of a sub-event are actively involved views, andtherefore determines that all actively involved views should receive aparticular sequence of sub-events. In other embodiments, even if touchsub-events were entirely confined to the area associated with oneparticular view, views higher in the hierarchy would still remain asactively involved views.

Event dispatcher module 174 dispatches the event information to an eventrecognizer (e.g., event recognizer 180). In embodiments including activeevent recognizer determination module 173, event dispatcher module 174delivers the event information to an event recognizer determined byactive event recognizer determination module 173. In some embodiments,event dispatcher module 174 stores in an event queue the eventinformation, which is retrieved by a respective event receiver module182.

In some embodiments, operating system 126 includes event sorter 170.Alternatively, application 136-1 includes event sorter 170. In yet otherembodiments, event sorter 170 is a stand-alone module, or a part ofanother module stored in memory 102, such as contact/motion module 130.

In some embodiments, application 136-1 includes a plurality of eventhandlers 190 and one or more application views 191, each of whichincludes instructions for handling touch events that occur within arespective view of the application's user interface. Each applicationview 191 of the application 136-1 includes one or more event recognizers180. Typically, a respective application view 191 includes a pluralityof event recognizers 180. In other embodiments, one or more of eventrecognizers 180 are part of a separate module, such as a user interfacekit (not shown) or a higher level object from which application 136-1inherits methods and other properties. In some embodiments, a respectiveevent handler 190 includes one or more of: data updater 176, objectupdater 177, GUI updater 178, and/or event data 179 received from eventsorter 170. Event handler 190 optionally utilizes or calls data updater176, object updater 177 or GUI updater 178 to update the applicationinternal state 192. Alternatively, one or more of the application views191 includes one or more respective event handlers 190. Also, in someembodiments, one or more of data updater 176, object updater 177, andGUI updater 178 are included in a respective application view 191.

A respective event recognizer 180 receives event information (e.g.,event data 179) from event sorter 170, and identifies an event from theevent information. Event recognizer 180 includes event receiver 182 andevent comparator 184. In some embodiments, event recognizer 180 alsoincludes at least a subset of: metadata 183, and event deliveryinstructions 188 (which optionally include sub-event deliveryinstructions).

Event receiver 182 receives event information from event sorter 170. Theevent information includes information about a sub-event, for example, atouch or a touch movement. Depending on the sub-event, the eventinformation also includes additional information, such as location ofthe sub-event. When the sub-event concerns motion of a touch, the eventinformation optionally also includes speed and direction of thesub-event. In some embodiments, events include rotation of the devicefrom one orientation to another (e.g., from a portrait orientation to alandscape orientation, or vice versa), and the event informationincludes corresponding information about the current orientation (alsocalled device attitude) of the device.

Event comparator 184 compares the event information to predefined eventor sub-event definitions and, based on the comparison, determines anevent or sub-event, or determines or updates the state of an event orsub-event. In some embodiments, event comparator 184 includes eventdefinitions 186. Event definitions 186 contain definitions of events(e.g., predefined sequences of sub-events), for example, event 1(187-1), event 2 (187-2), and others. In some embodiments, sub-events inan event 187 include, for example, touch begin, touch end, touchmovement, touch cancellation, and multiple touching. In one example, thedefinition for event 1 (187-1) is a double tap on a displayed object.The double tap, for example, comprises a first touch (touch begin) onthe displayed object for a predetermined phase, a first lift-off (touchend) for a predetermined phase, a second touch (touch begin) on thedisplayed object for a predetermined phase, and a second lift-off (touchend) for a predetermined phase. In another example, the definition forevent 2 (187-2) is a dragging on a displayed object. The dragging, forexample, comprises a touch (or contact) on the displayed object for apredetermined phase, a movement of the touch across touch-sensitivedisplay system 112, and lift-off of the touch (touch end). In someembodiments, the event also includes information for one or moreassociated event handlers 190.

In some embodiments, event definition 187 includes a definition of anevent for a respective user-interface object. In some embodiments, eventcomparator 184 performs a hit test to determine which user-interfaceobject is associated with a sub-event. For example, in an applicationview in which three user-interface objects are displayed ontouch-sensitive display system 112, when a touch is detected ontouch-sensitive display system 112, event comparator 184 performs a hittest to determine which of the three user-interface objects isassociated with the touch (sub-event). If each displayed object isassociated with a respective event handler 190, the event comparatoruses the result of the hit test to determine which event handler 190should be activated. For example, event comparator 184 selects an eventhandler associated with the sub-event and the object triggering the hittest.

In some embodiments, the definition for a respective event 187 alsoincludes delayed actions that delay delivery of the event informationuntil after it has been determined whether the sequence of sub-eventsdoes or does not correspond to the event recognizer's event type.

When a respective event recognizer 180 determines that the series ofsub-events do not match any of the events in event definitions 186, therespective event recognizer 180 enters an event impossible, eventfailed, or event ended state, after which it disregards subsequentsub-events of the touch-based gesture. In this situation, other eventrecognizers, if any, that remain active for the hit view continue totrack and process sub-events of an ongoing touch-based gesture.

In some embodiments, a respective event recognizer 180 includes metadata183 with configurable properties, flags, and/or lists that indicate howthe event delivery system should perform sub-event delivery to activelyinvolved event recognizers. In some embodiments, metadata 183 includesconfigurable properties, flags, and/or lists that indicate how eventrecognizers interact, or are enabled to interact, with one another. Insome embodiments, metadata 183 includes configurable properties, flags,and/or lists that indicate whether sub-events are delivered to varyinglevels in the view or programmatic hierarchy.

In some embodiments, a respective event recognizer 180 activates eventhandler 190 associated with an event when one or more particularsub-events of an event are recognized. In some embodiments, a respectiveevent recognizer 180 delivers event information associated with theevent to event handler 190. Activating an event handler 190 is distinctfrom sending (and deferred sending) sub-events to a respective hit view.In some embodiments, event recognizer 180 throws a flag associated withthe recognized event, and event handler 190 associated with the flagcatches the flag and performs a predefined process.

In some embodiments, event delivery instructions 188 include sub-eventdelivery instructions that deliver event information about a sub-eventwithout activating an event handler. Instead, the sub-event deliveryinstructions deliver event information to event handlers associated withthe series of sub-events or to actively involved views. Event handlersassociated with the series of sub-events or with actively involved viewsreceive the event information and perform a predetermined process.

In some embodiments, data updater 176 creates and updates data used inapplication 136-1. For example, data updater 176 updates the telephonenumber used in contacts module 137, or stores a video file used in videoplayer module 145. In some embodiments, object updater 177 creates andupdates objects used in application 136-1. For example, object updater177 creates a new user-interface object or updates the position of auser-interface object. GUI updater 178 updates the GUI. For example, GUIupdater 178 prepares display information and sends it to graphics module132 for display on a touch-sensitive display.

In some embodiments, event handler(s) 190 includes or has access to dataupdater 176, object updater 177, and GUI updater 178. In someembodiments, data updater 176, object updater 177, and GUI updater 178are included in a single module of a respective application 136-1 orapplication view 191. In other embodiments, they are included in two ormore software modules.

It shall be understood that the foregoing discussion regarding eventhandling of user touches on touch-sensitive displays also applies toother forms of user inputs to operate multifunction devices 100 withinput-devices, not all of which are initiated on touch screens. Forexample, mouse movement and mouse button presses, optionally coordinatedwith single or multiple keyboard presses or holds; contact movementssuch as taps, drags, scrolls, etc., on touch-pads; pen stylus inputs;movement of the device; oral instructions; detected eye movements;biometric inputs; and/or any combination thereof are optionally utilizedas inputs corresponding to sub-events which define an event to berecognized.

FIG. 1C is a block diagram illustrating a tactile output module inaccordance with some embodiments. In some embodiments, I/O subsystem 106(e.g., haptic feedback controller 161 (FIG. 1A) and/or other inputcontroller(s) 160 (FIG. 1A)) includes at least some of the examplecomponents shown in FIG. 1C. In some embodiments, peripherals interface118 includes at least some of the example components shown in FIG. 1C.

In some embodiments, the tactile output module includes haptic feedbackmodule 133. In some embodiments, haptic feedback module 133 aggregatesand combines tactile outputs for user interface feedback from softwareapplications on the electronic device (e.g., feedback that is responsiveto user inputs that correspond to displayed user interfaces and alertsand other notifications that indicate the performance of operations oroccurrence of events in user interfaces of the electronic device).Haptic feedback module 133 includes one or more of: waveform module 123(for providing waveforms used for generating tactile outputs), mixer 125(for mixing waveforms, such as waveforms in different channels),compressor 127 (for reducing or compressing a dynamic range of thewaveforms), low-pass filter 129 (for filtering out high frequency signalcomponents in the waveforms), and thermal controller 131 (for adjustingthe waveforms in accordance with thermal conditions). In someembodiments, haptic feedback module 133 is included in haptic feedbackcontroller 161 (FIG. 1A). In some embodiments, a separate unit of hapticfeedback module 133 (or a separate implementation of haptic feedbackmodule 133) is also included in an audio controller (e.g., audiocircuitry 110, FIG. 1A) and used for generating audio signals. In someembodiments, a single haptic feedback module 133 is used for generatingaudio signals and generating waveforms for tactile outputs.

In some embodiments, haptic feedback module 133 also includes triggermodule 121 (e.g., a software application, operating system, or othersoftware module that determines a tactile output is to be generated andinitiates the process for generating the corresponding tactile output).In some embodiments, trigger module 121 generates trigger signals forinitiating generation of waveforms (e.g., by waveform module 123). Forexample, trigger module 121 generates trigger signals based on presettiming criteria. In some embodiments, trigger module 121 receivestrigger signals from outside haptic feedback module 133 (e.g., in someembodiments, haptic feedback module 133 receives trigger signals fromhardware input processing module 146 located outside haptic feedbackmodule 133) and relays the trigger signals to other components withinhaptic feedback module 133 (e.g., waveform module 123) or softwareapplications that trigger operations (e.g., with trigger module 121)based on activation of a user interface element (e.g., an applicationicon or an affordance within an application) or a hardware input device(e.g., a home button). In some embodiments, trigger module 121 alsoreceives tactile feedback generation instructions (e.g., from hapticfeedback module 133, FIGS. 1A and 3). In some embodiments, triggermodule 121 generates trigger signals in response to haptic feedbackmodule 133 (or trigger module 121 in haptic feedback module 133)receiving tactile feedback instructions (e.g., from haptic feedbackmodule 133, FIGS. 1A and 3).

Waveform module 123 receives trigger signals (e.g., from trigger module121) as an input, and in response to receiving trigger signals, provideswaveforms for generation of one or more tactile outputs (e.g., waveformsselected from a predefined set of waveforms designated for use bywaveform module 123, such as the waveforms described in greater detailbelow with reference to FIGS. 4F-4G).

Mixer 125 receives waveforms (e.g., from waveform module 123) as aninput, and mixes together the waveforms. For example, when mixer 125receives two or more waveforms (e.g., a first waveform in a firstchannel and a second waveform that at least partially overlaps with thefirst waveform in a second channel) mixer 125 outputs a combinedwaveform that corresponds to a sum of the two or more waveforms. In someembodiments, mixer 125 also modifies one or more waveforms of the two ormore waveforms to emphasize particular waveform(s) over the rest of thetwo or more waveforms (e.g., by increasing a scale of the particularwaveform(s) and/or decreasing a scale of the rest of the waveforms). Insome circumstances, mixer 125 selects one or more waveforms to removefrom the combined waveform (e.g., the waveform from the oldest source isdropped when there are waveforms from more than three sources that havebeen requested to be output concurrently by tactile output generator167).

Compressor 127 receives waveforms (e.g., a combined waveform from mixer125) as an input, and modifies the waveforms. In some embodiments,compressor 127 reduces the waveforms (e.g., in accordance with physicalspecifications of tactile output generators 167 (FIG. 1A) or 357 (FIG.3)) so that tactile outputs corresponding to the waveforms are reduced.In some embodiments, compressor 127 limits the waveforms, such as byenforcing a predefined maximum amplitude for the waveforms. For example,compressor 127 reduces amplitudes of portions of waveforms that exceed apredefined amplitude threshold while maintaining amplitudes of portionsof waveforms that do not exceed the predefined amplitude threshold. Insome embodiments, compressor 127 reduces a dynamic range of thewaveforms. In some embodiments, compressor 127 dynamically reduces thedynamic range of the waveforms so that the combined waveforms remainwithin performance specifications of the tactile output generator 167(e.g., force and/or movable mass displacement limits).

Low-pass filter 129 receives waveforms (e.g., compressed waveforms fromcompressor 127) as an input, and filters (e.g., smooths) the waveforms(e.g., removes or reduces high frequency signal components in thewaveforms). For example, in some instances, compressor 127 includes, incompressed waveforms, extraneous signals (e.g., high frequency signalcomponents) that interfere with the generation of tactile outputs and/orexceed performance specifications of tactile output generator 167 whenthe tactile outputs are generated in accordance with the compressedwaveforms. Low-pass filter 129 reduces or removes such extraneoussignals in the waveforms.

Thermal controller 131 receives waveforms (e.g., filtered waveforms fromlow-pass filter 129) as an input, and adjusts the waveforms inaccordance with thermal conditions of device 100 (e.g., based oninternal temperatures detected within device 100, such as thetemperature of haptic feedback controller 161, and/or externaltemperatures detected by device 100). For example, in some cases, theoutput of haptic feedback controller 161 varies depending on thetemperature (e.g. haptic feedback controller 161, in response toreceiving same waveforms, generates a first tactile output when hapticfeedback controller 161 is at a first temperature and generates a secondtactile output when haptic feedback controller 161 is at a secondtemperature that is distinct from the first temperature). For example,the magnitude (or the amplitude) of the tactile outputs may varydepending on the temperature. To reduce the effect of the temperaturevariations, the waveforms are modified (e.g., an amplitude of thewaveforms is increased or decreased based on the temperature).

In some embodiments, haptic feedback module 133 (e.g., trigger module121) is coupled to hardware input processing module 146. In someembodiments, other input controller(s) 160 in FIG. 1A includes hardwareinput processing module 146. In some embodiments, hardware inputprocessing module 146 receives inputs from hardware input device 145(e.g., other input or control devices 116 in FIG. 1A, such as a homebutton). In some embodiments, hardware input device 145 is any inputdevice described herein, such as touch-sensitive display system 112(FIG. 1A), keyboard/mouse 350 (FIG. 3), touchpad 355 (FIG. 3), one ofother input or control devices 116 (FIG. 1A), or an intensity-sensitivehome button (e.g., as shown in FIG. 2B or a home button with amechanical actuator as illustrated in FIG. 2C). In some embodiments,hardware input device 145 consists of an intensity-sensitive home button(e.g., as shown in FIG. 2B or a home button with a mechanical actuatoras illustrated in FIG. 2C), and not touch-sensitive display system 112(FIG. 1A), keyboard/mouse 350 (FIG. 3), or touchpad 355 (FIG. 3). Insome embodiments, in response to inputs from hardware input device 145,hardware input processing module 146 provides one or more triggersignals to haptic feedback module 133 to indicate that a user inputsatisfying predefined input criteria, such as an input corresponding toa “click” of a home button (e.g., a “down click” or an “up click”), hasbeen detected. In some embodiments, haptic feedback module 133 provideswaveforms that correspond to the “click” of a home button in response tothe input corresponding to the “click” of a home button, simulating ahaptic feedback of pressing a physical home button.

In some embodiments, the tactile output module includes haptic feedbackcontroller 161 (e.g., haptic feedback controller 161 in FIG. 1A), whichcontrols the generation of tactile outputs. In some embodiments, hapticfeedback controller 161 is coupled to a plurality of tactile outputgenerators, and selects one or more tactile output generators of theplurality of tactile output generators and sends waveforms to theselected one or more tactile output generators for generating tactileoutputs. In some embodiments, haptic feedback controller 161 coordinatestactile output requests that correspond to activation of hardware inputdevice 145 and tactile output requests that correspond to softwareevents (e.g., tactile output requests from haptic feedback module 133)and modifies one or more waveforms of the two or more waveforms toemphasize particular waveform(s) over the rest of the two or morewaveforms (e.g., by increasing a scale of the particular waveform(s)and/or decreasing a scale of the rest of the waveforms, such as toprioritize tactile outputs that correspond to activations of hardwareinput device 145 over tactile outputs that correspond to softwareevents).

In some embodiments, as shown in FIG. 1C, an output of haptic feedbackcontroller 161 is coupled to audio circuitry of device 100 (e.g., audiocircuitry 110, FIG. 1A), and provides audio signals to audio circuitryof device 100. In some embodiments, haptic feedback controller 161provides both waveforms used for generating tactile outputs and audiosignals used for providing audio outputs in conjunction with generationof the tactile outputs. In some embodiments, haptic feedback controller161 modifies audio signals and/or waveforms (used for generating tactileoutputs) so that the audio outputs and the tactile outputs aresynchronized (e.g., by delaying the audio signals and/or waveforms). Insome embodiments, haptic feedback controller 161 includes adigital-to-analog converter used for converting digital waveforms intoanalog signals, which are received by amplifier 163 and/or tactileoutput generator 167.

In some embodiments, the tactile output module includes amplifier 163.In some embodiments, amplifier 163 receives waveforms (e.g., from hapticfeedback controller 161) and amplifies the waveforms prior to sendingthe amplified waveforms to tactile output generator 167 (e.g., any oftactile output generators 167 (FIG. 1A) or 357 (FIG. 3)). For example,amplifier 163 amplifies the received waveforms to signal levels that arein accordance with physical specifications of tactile output generator167 (e.g., to a voltage and/or a current required by tactile outputgenerator 167 for generating tactile outputs so that the signals sent totactile output generator 167 produce tactile outputs that correspond tothe waveforms received from haptic feedback controller 161) and sendsthe amplified waveforms to tactile output generator 167. In response,tactile output generator 167 generates tactile outputs (e.g., byshifting a movable mass back and forth in one or more dimensionsrelative to a neutral position of the movable mass).

In some embodiments, the tactile output module includes sensor 169,which is coupled to tactile output generator 167. Sensor 169 detectsstates or state changes (e.g., mechanical position, physicaldisplacement, and/or movement) of tactile output generator 167 or one ormore components of tactile output generator 167 (e.g., one or moremoving parts, such as a membrane, used to generate tactile outputs). Insome embodiments, sensor 169 is a magnetic field sensor (e.g., a Halleffect sensor) or other displacement and/or movement sensor. In someembodiments, sensor 169 provides information (e.g., a position, adisplacement, and/or a movement of one or more parts in tactile outputgenerator 167) to haptic feedback controller 161 and, in accordance withthe information provided by sensor 169 about the state of tactile outputgenerator 167, haptic feedback controller 161 adjusts the waveformsoutput from haptic feedback controller 161 (e.g., waveforms sent totactile output generator 167, optionally via amplifier 163).

FIGS. 1D-1E are block diagrams illustrating application-independentsoftware modules in accordance with some embodiments.

FIG. 1D illustrates that haptic feedback module 133 receives informationrelated to user interactions with device 100. Haptic feedback module133, in response to receiving the information, generates and providestactile feedback generation instructions to haptic feedback controller161, which in turn sends instructions to tactile output generators 167for generating tactile outputs. In some embodiments, haptic feedbackmodule 133 corresponds to a synthesizer illustrated in Appendix A and/oran AVHapticPlayer in Appendix B.

In some embodiments, applications (e.g., application 1 (136-1)) sendinformation (e.g., information 272) related to user interactions withdevice 100 to application-independent software module 260, whichprocesses the information and sends the processed information (e.g.,information 274, which is distinct from information 272) to hapticfeedback module 133. In some embodiments, applications correspond toClient App illustrated in Appendices A and B.

In some embodiments, application-independent software module 260 isdistinct and separate from applications, as shown in FIG. 1D. In someembodiments, application-independent software module 260 includes theUIKit, and more particularly, the UIFeedbackGenerator in the UIKit,illustrated in Appendices A and B.

In some embodiments, applications use various functions and objectsdescribed in Appendices A and B to communicate withapplication-independent software module 260.

FIG. 1E is similar to FIG. 1D, except that each of application 1 (136-1,such as an e-mail client application) and application 2 (136-2, such asa browser application) includes a distinct instance (e.g., a distinctcopy) of application-independent software module 260. In addition, eachof the applications (e.g., application 1 and application 2) includes anapplication core that is specific to the application (e.g., application1 (136-1) includes application core 1 (270-1) and/or application 2(136-2) includes application core 2 (270-2)). For example, applicationcore 1 (270-1) includes instructions for performing operations specificto application 1 (136-1) (e.g., retrieving e-mails from one or moree-mail servers) and application core 2 (260-2) includes instructions forperforming operations specific to application 2 (136-2) (e.g.,bookmarking a web page).

FIG. 1E also illustrates that an application core (e.g., applicationcore 1 (270-1) receives user interaction models from applicationindependent software module 260 or an instance of applicationindependent software module 260. In some embodiments, the userinteraction models include information identifying a plurality of userinterface element types 272 (e.g., slider objects 272-1, switch objects272-2, etc.) and associated user interface responses 274. In someembodiments, the user interface responses include informationidentifying tactile outputs to be generated in conjunction with userinteraction with corresponding user interface elements (e.g., sliderobjects represented by slider object type 272-1 are associated withtactile output 274-1, and switch objects represented by switch objecttype 272-2 are associated with tactile output 274-2). For example,tactile output 274-1 defines which tactile outputs are to be generatedwhen a corresponding slider thumb is moved and/or when the slider thumbreaches an end of the slider, and tactile output 274-2 defines whichtactile outputs are to be generated when a corresponding switch objectis turned on and/or off.

In some embodiments, user interaction models are predefined classes(e.g., the UIView class defining an area on a display and the interfacesfor managing the content in that area). At run time, a view object isprepared (e.g., an instance of the UIView class is created) to handlethe rendering of content in a corresponding area and also handleinteractions with the content. In some embodiments, the UIView classincludes one or more subclasses, such as a UIControl class configuredfor implementing visual elements that are used for specific actions, aUITableView class configured for displaying and editing hierarchicallists of information, a UIDatePicker class configured for displaying awheel of time to allow a user to select dates and/or times, a UISliderclass configured for selecting a single value from a continuous range ofvalues, and a UIScrollView class configured for displaying content thatis larger than a corresponding display area. In some embodiments, theUITableView class, the UIDatePicker class, the UISlider class, and theUIScrollView class are subclasses of the UIControl class.

FIG. 2A illustrates a portable multifunction device 100 having a touchscreen (e.g., touch-sensitive display system 112, FIG. 1A) in accordancewith some embodiments. The touch screen optionally displays one or moregraphics within user interface (UI) 200. In these embodiments, as wellas others described below, a user is enabled to select one or more ofthe graphics by making a gesture on the graphics, for example, with oneor more fingers 202 (not drawn to scale in the figure) or one or morestyluses 203 (not drawn to scale in the figure). In some embodiments,selection of one or more graphics occurs when the user breaks contactwith the one or more graphics. In some embodiments, the gestureoptionally includes one or more taps, one or more swipes (from left toright, right to left, upward and/or downward) and/or a rolling of afinger (from right to left, left to right, upward and/or downward) thathas made contact with device 100. In some implementations orcircumstances, inadvertent contact with a graphic does not select thegraphic. For example, a swipe gesture that sweeps over an applicationicon optionally does not select the corresponding application when thegesture corresponding to selection is a tap.

Device 100 optionally also includes one or more physical buttons, suchas “home” or menu button 204. As described previously, menu button 204is, optionally, used to navigate to any application 136 in a set ofapplications that are, optionally executed on device 100. Alternatively,in some embodiments, the menu button is implemented as a soft key in aGUI displayed on the touch-screen display.

In some embodiments, device 100 includes the touch-screen display, menubutton 204 (sometimes called home button 204), push button 206 forpowering the device on/off and locking the device, volume adjustmentbutton(s) 208, Subscriber Identity Module (SIM) card slot 210, head setjack 212, and docking/charging external port 124. Push button 206 is,optionally, used to turn the power on/off on the device by depressingthe button and holding the button in the depressed state for apredefined time interval; to lock the device by depressing the buttonand releasing the button before the predefined time interval haselapsed; and/or to unlock the device or initiate an unlock process. Insome embodiments, device 100 also accepts verbal input for activation ordeactivation of some functions through microphone 113. Device 100 also,optionally, includes one or more contact intensity sensors 165 fordetecting intensity of contacts on touch-sensitive display system 112and/or one or more tactile output generators 167 for generating tactileoutputs for a user of device 100.

In some embodiments, an application (e.g., an application core of theapplication, such as application core 1 (270-1)) generates a userinterface by populating a user interface element (e.g., a slider)identified in the user interaction models with content provided by theapplication core (e.g., application core 1 (270-1) provides a range ofthe slider element and locations and/or values of major and minor tickmarks).

In some embodiments, the application sends the user interface tographics module 132 for display.

In some embodiments, the application receives information representinguser interactions with device 100 from contact/motion module 130. Insome embodiments, the application relays the information representinguser interactions with device 100 to an application-independent softwaremodule or an instance thereof (e.g., 260-1), and theapplication-independent software module send the information or a subsetthereof to haptic feedback module 133 for generating tactile outputs. Insome embodiments, the application also sends information identifying oneor more dimensions of a selected user interface. In some embodiments,the information representing user interactions with device 100 includesa magnitude (e.g., a speed) and/or a location of a user input.

In some embodiments, haptic feedback module 133 receives the informationrepresenting user interactions with device 100 directly fromcontact/motion module 130 (e.g., without going through the application).

FIG. 3 is a block diagram of an example multifunction device with adisplay and a touch-sensitive surface in accordance with someembodiments. Device 300 need not be portable. In some embodiments,device 300 is a laptop computer, a desktop computer, a tablet computer,a multimedia player device, a navigation device, an educational device(such as a child's learning toy), a gaming system, or a control device(e.g., a home or industrial controller). Device 300 typically includesone or more processing units (CPU's) 310, one or more network or othercommunications interfaces 360, memory 370, and one or more communicationbuses 320 for interconnecting these components. Communication buses 320optionally include circuitry (sometimes called a chipset) thatinterconnects and controls communications between system components.Device 300 includes input/output (I/O) interface 330 comprising display340, which is typically a touch-screen display. I/O interface 330 alsooptionally includes a keyboard and/or mouse (or other pointing device)350 and touchpad 355, tactile output generator 357 for generatingtactile outputs on device 300 (e.g., similar to tactile outputgenerator(s) 167 described above with reference to FIG. 1A), sensors 359(e.g., optical, acceleration, proximity, touch-sensitive, and/or contactintensity sensors similar to contact intensity sensor(s) 165 describedabove with reference to FIG. 1A). Memory 370 includes high-speed randomaccess memory, such as DRAM, SRAM, DDR RAM or other random access solidstate memory devices; and optionally includes non-volatile memory, suchas one or more magnetic disk storage devices, optical disk storagedevices, flash memory devices, or other non-volatile solid state storagedevices. Memory 370 optionally includes one or more storage devicesremotely located from CPU(s) 310. In some embodiments, memory 370 storesprograms, modules, and data structures analogous to the programs,modules, and data structures stored in memory 102 of portablemultifunction device 100 (FIG. 1A), or a subset thereof. Furthermore,memory 370 optionally stores additional programs, modules, and datastructures not present in memory 102 of portable multifunction device100. For example, memory 370 of device 300 optionally stores drawingmodule 380, presentation module 382, word processing module 384, website creation module 386, disk authoring module 388, and/or spreadsheetmodule 390, while memory 102 of portable multifunction device 100 (FIG.1A) optionally does not store these modules.

Each of the above identified elements in FIG. 3 are, optionally, storedin one or more of the previously mentioned memory devices. Each of theabove identified modules corresponds to a set of instructions forperforming a function described above. The above identified modules orprograms (i.e., sets of instructions) need not be implemented asseparate software programs, procedures or modules, and thus varioussubsets of these modules are, optionally, combined or otherwisere-arranged in various embodiments. In some embodiments, memory 370optionally stores a subset of the modules and data structures identifiedabove. Furthermore, memory 370 optionally stores additional modules anddata structures not described above.

Attention is now directed towards embodiments of user interfaces (“UI”)that are, optionally, implemented on portable multifunction device 100.

FIG. 4A illustrates an example user interface for a menu of applicationson portable multifunction device 100 in accordance with someembodiments. Similar user interfaces are, optionally, implemented ondevice 300. In some embodiments, user interface 400 includes thefollowing elements, or a subset or superset thereof:

-   Signal strength indicator(s) 402 for wireless communication(s), such    as cellular and Wi-Fi signals;-   Time 404;-   a Bluetooth indicator;-   Battery status indicator 406;-   Tray 408 with icons for frequently used applications, such as:    -   Icon 416 for telephone module 138, labeled “Phone,” which        optionally includes an indicator 414 of the number of missed        calls or voicemail messages;    -   Icon 418 for e-mail client module 140, labeled “Mail,” which        optionally includes an indicator 410 of the number of unread        e-mails;    -   Icon 420 for browser module 147, labeled “Browser;” and    -   Icon 422 for video and music player module 152, also referred to        as iPod (trademark of Apple Inc.) module 152, labeled “iPod;”        and    -   Icons for other applications, such as:        -   Icon 424 for IM module 141, labeled “Messages;”        -   Icon 426 for calendar module 148, labeled “Calendar;”        -   Icon 428 for image management module 144, labeled “Photos;”        -   Icon 430 for camera module 143, labeled “Camera;”        -   Icon 432 for online video module 155, labeled “Online            Video;”        -   Icon 434 for stocks widget 149-2, labeled “Stocks;”        -   Icon 436 for map module 154, labeled “Maps;”        -   Icon 438 for weather widget 149-1, labeled “Weather;”        -   Icon 440 for alarm clock widget 149-4, labeled “Clock;”        -   Icon 442 for workout support module 142, labeled “Workout            Support;”        -   Icon 444 for notes module 153, labeled “Notes;” and        -   Icon 446 for a settings application or module, which            provides access to settings for device 100 and its various            applications 136.

It should be noted that the icon labels illustrated in FIG. 4A aremerely examples. For example, in some embodiments, icon 422 for videoand music player module 152 is labeled “Music” or “Music Player.” Otherlabels are, optionally, used for various application icons. In someembodiments, a label for a respective application icon includes a nameof an application corresponding to the respective application icon. Insome embodiments, a label for a particular application icon is distinctfrom a name of an application corresponding to the particularapplication icon.

FIG. 4B illustrates an example user interface on a device (e.g., device300, FIG. 3) with a touch-sensitive surface 451 (e.g., a tablet ortouchpad 355, FIG. 3) that is separate from the display 450. Device 300also, optionally, includes one or more contact intensity sensors (e.g.,one or more of sensors 357) for detecting intensity of contacts ontouch-sensitive surface 451 and/or one or more tactile output generators359 for generating tactile outputs for a user of device 300.

Although many of the examples that follow will be given with referenceto inputs on touch screen display 112 (where the touch sensitive surfaceand the display are combined), in some embodiments, the device detectsinputs on a touch-sensitive surface that is separate from the display,as shown in FIG. 4B. In some embodiments, the touch-sensitive surface(e.g., 451 in FIG. 4B) has a primary axis (e.g., 452 in FIG. 4B) thatcorresponds to a primary axis (e.g., 453 in FIG. 4B) on the display(e.g., 450). In accordance with these embodiments, the device detectscontacts (e.g., 460 and 462 in FIG. 4B) with the touch-sensitive surface451 at locations that correspond to respective locations on the display(e.g., in FIG. 4B, 460 corresponds to 468 and 462 corresponds to 470).In this way, user inputs (e.g., contacts 460 and 462, and movementsthereof) detected by the device on the touch-sensitive surface (e.g.,451 in FIG. 4B) are used by the device to manipulate the user interfaceon the display (e.g., 450 in FIG. 4B) of the multifunction device whenthe touch-sensitive surface is separate from the display. It should beunderstood that similar methods are, optionally, used for other userinterfaces described herein.

Additionally, while the following examples are given primarily withreference to finger inputs (e.g., finger contacts, finger tap gestures,finger swipe gestures, etc.), it should be understood that, in someembodiments, one or more of the finger inputs are replaced with inputfrom another input device (e.g., a stylus input). Similarly, whenmultiple user inputs are simultaneously detected, it should beunderstood that multiple finger contacts, or a combination of fingercontacts and stylus inputs are used simultaneously.

As used in the specification and claims, the term “intensity” of acontact on a touch-sensitive surface refers to the force or pressure(force per unit area) of a contact (e.g., a finger contact or a styluscontact) on the touch-sensitive surface, or to a substitute (proxy) forthe force or pressure of a contact on the touch-sensitive surface. Theintensity of a contact has a range of values that includes at least fourdistinct values and more typically includes hundreds of distinct values(e.g., at least 256). Intensity of a contact is, optionally, determined(or measured) using various approaches and various sensors orcombinations of sensors. For example, one or more force sensorsunderneath or adjacent to the touch-sensitive surface are, optionally,used to measure force at various points on the touch-sensitive surface.In some implementations, force measurements from multiple force sensorsare combined (e.g., a weighted average or a sum) to determine anestimated force of a contact. Similarly, a pressure-sensitive tip of astylus is, optionally, used to determine a pressure of the stylus on thetouch-sensitive surface. Alternatively, the size of the contact areadetected on the touch-sensitive surface and/or changes thereto, thecapacitance of the touch-sensitive surface proximate to the contactand/or changes thereto, and/or the resistance of the touch-sensitivesurface proximate to the contact and/or changes thereto are, optionally,used as a substitute for the force or pressure of the contact on thetouch-sensitive surface. In some implementations, the substitutemeasurements for contact force or pressure are used directly todetermine whether an intensity threshold has been exceeded (e.g., theintensity threshold is described in units corresponding to thesubstitute measurements). In some implementations, the substitutemeasurements for contact force or pressure are converted to an estimatedforce or pressure and the estimated force or pressure is used todetermine whether an intensity threshold has been exceeded (e.g., theintensity threshold is a pressure threshold measured in units ofpressure). Using the intensity of a contact as an attribute of a userinput allows for user access to additional device functionality that mayotherwise not be readily accessible by the user on a reduced-size devicewith limited real estate for displaying affordances (e.g., on atouch-sensitive display) and/or receiving user input (e.g., via atouch-sensitive display, a touch-sensitive surface, or aphysical/mechanical control such as a knob or a button).

In some embodiments, contact/motion module 130 uses a set of one or moreintensity thresholds to determine whether an operation has beenperformed by a user (e.g., to determine whether a user has “clicked” onan icon). In some embodiments, at least a subset of the intensitythresholds are determined in accordance with software parameters (e.g.,the intensity thresholds are not determined by the activation thresholdsof particular physical actuators and can be adjusted without changingthe physical hardware of device 100). For example, a mouse “click”threshold of a trackpad or touch-screen display can be set to any of alarge range of predefined thresholds values without changing thetrackpad or touch-screen display hardware. Additionally, in someimplementations a user of the device is provided with software settingsfor adjusting one or more of the set of intensity thresholds (e.g., byadjusting individual intensity thresholds and/or by adjusting aplurality of intensity thresholds at once with a system-level click“intensity” parameter).

As used in the specification and claims, the term “characteristicintensity” of a contact refers to a characteristic of the contact basedon one or more intensities of the contact. In some embodiments, thecharacteristic intensity is based on multiple intensity samples. Thecharacteristic intensity is, optionally, based on a predefined number ofintensity samples, or a set of intensity samples collected during apredetermined time period (e.g., 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10seconds) relative to a predefined event (e.g., after detecting thecontact, prior to detecting liftoff of the contact, before or afterdetecting a start of movement of the contact, prior to detecting an endof the contact, before or after detecting an increase in intensity ofthe contact, and/or before or after detecting a decrease in intensity ofthe contact). A characteristic intensity of a contact is, optionallybased on one or more of: a maximum value of the intensities of thecontact, a mean value of the intensities of the contact, an averagevalue of the intensities of the contact, a top 10 percentile value ofthe intensities of the contact, a value at the half maximum of theintensities of the contact, a value at the 90 percent maximum of theintensities of the contact, a value produced by low-pass filtering theintensity of the contact over a predefined period or starting at apredefined time, or the like. In some embodiments, the duration of thecontact is used in determining the characteristic intensity (e.g., whenthe characteristic intensity is an average of the intensity of thecontact over time). In some embodiments, the characteristic intensity iscompared to a set of one or more intensity thresholds to determinewhether an operation has been performed by a user. For example, the setof one or more intensity thresholds may include a first intensitythreshold and a second intensity threshold. In this example, a contactwith a characteristic intensity that does not exceed the first intensitythreshold results in a first operation, a contact with a characteristicintensity that exceeds the first intensity threshold and does not exceedthe second intensity threshold results in a second operation, and acontact with a characteristic intensity that exceeds the secondintensity threshold results in a third operation. In some embodiments, acomparison between the characteristic intensity and one or moreintensity thresholds is used to determine whether or not to perform oneor more operations (e.g., whether to perform a respective option orforgo performing the respective operation) rather than being used todetermine whether to perform a first operation or a second operation.

In some embodiments, a portion of a gesture is identified for purposesof determining a characteristic intensity. For example, atouch-sensitive surface may receive a continuous swipe contacttransitioning from a start location and reaching an end location (e.g.,a drag gesture), at which point the intensity of the contact increases.In this example, the characteristic intensity of the contact at the endlocation may be based on only a portion of the continuous swipe contact,and not the entire swipe contact (e.g., only the portion of the swipecontact at the end location). In some embodiments, a smoothing algorithmmay be applied to the intensities of the swipe contact prior todetermining the characteristic intensity of the contact. For example,the smoothing algorithm optionally includes one or more of: anunweighted sliding-average smoothing algorithm, a triangular smoothingalgorithm, a median filter smoothing algorithm, and/or an exponentialsmoothing algorithm. In some circumstances, these smoothing algorithmseliminate narrow spikes or dips in the intensities of the swipe contactfor purposes of determining a characteristic intensity.

The user interface figures described herein optionally include variousintensity diagrams that show the current intensity of the contact on thetouch-sensitive surface relative to one or more intensity thresholds(e.g., a contact detection intensity threshold IT₀, a light pressintensity threshold IT_(L), a deep press intensity threshold IT_(D)(e.g., that is at least initially higher than IT_(L)), and/or one ormore other intensity thresholds (e.g., an intensity threshold IT_(H)that is lower than IT_(L))). This intensity diagram is typically notpart of the displayed user interface, but is provided to aid in theinterpretation of the figures. In some embodiments, the light pressintensity threshold corresponds to an intensity at which the device willperform operations typically associated with clicking a button of aphysical mouse or a trackpad. In some embodiments, the deep pressintensity threshold corresponds to an intensity at which the device willperform operations that are different from operations typicallyassociated with clicking a button of a physical mouse or a trackpad. Insome embodiments, when a contact is detected with a characteristicintensity below the light press intensity threshold (e.g., and above anominal contact-detection intensity threshold IT₀ below which thecontact is no longer detected), the device will move a focus selector inaccordance with movement of the contact on the touch-sensitive surfacewithout performing an operation associated with the light pressintensity threshold or the deep press intensity threshold. Generally,unless otherwise stated, these intensity thresholds are consistentbetween different sets of user interface figures.

In some embodiments, the response of the device to inputs detected bythe device depends on criteria based on the contact intensity during theinput. For example, for some “light press” inputs, the intensity of acontact exceeding a first intensity threshold during the input triggersa first response. In some embodiments, the response of the device toinputs detected by the device depends on criteria that include both thecontact intensity during the input and time-based criteria. For example,for some “deep press” inputs, the intensity of a contact exceeding asecond intensity threshold during the input, greater than the firstintensity threshold for a light press, triggers a second response onlyif a delay time has elapsed between meeting the first intensitythreshold and meeting the second intensity threshold. This delay time istypically less than 200 ms (milliseconds) in duration (e.g., 40, 100, or120 ms, depending on the magnitude of the second intensity threshold,with the delay time increasing as the second intensity thresholdincreases). This delay time helps to avoid accidental recognition ofdeep press inputs. As another example, for some “deep press” inputs,there is a reduced-sensitivity time period that occurs after the time atwhich the first intensity threshold is met. During thereduced-sensitivity time period, the second intensity threshold isincreased. This temporary increase in the second intensity thresholdalso helps to avoid accidental deep press inputs. For other deep pressinputs, the response to detection of a deep press input does not dependon time-based criteria.

In some embodiments, one or more of the input intensity thresholdsand/or the corresponding outputs vary based on one or more factors, suchas user settings, contact motion, input timing, application running,rate at which the intensity is applied, number of concurrent inputs,user history, environmental factors (e.g., ambient noise), focusselector position, and the like. Example factors are described in U.S.patent application Ser. Nos. 14/399,606 and 14/624,296, which areincorporated by reference herein in their entireties.

For example, FIG. 4C illustrates a dynamic intensity threshold 480 thatchanges over time based in part on the intensity of touch input 476 overtime. Dynamic intensity threshold 480 is a sum of two components, firstcomponent 474 that decays over time after a predefined delay time plfrom when touch input 476 is initially detected, and second component478 that trails the intensity of touch input 476 over time. The initialhigh intensity threshold of first component 474 reduces accidentaltriggering of a “deep press” response, while still allowing an immediate“deep press” response if touch input 476 provides sufficient intensity.Second component 478 reduces unintentional triggering of a “deep press”response by gradual intensity fluctuations of in a touch input. In someembodiments, when touch input 476 satisfies dynamic intensity threshold480 (e.g., at point 481 in FIG. 4C), the “deep press” response istriggered.

FIG. 4D illustrates another dynamic intensity threshold 486 (e.g.,intensity threshold I_(D)). FIG. 4D also illustrates two other intensitythresholds: a first intensity threshold I_(H) and a second intensitythreshold I_(L). In FIG. 4D, although touch input 484 satisfies thefirst intensity threshold I_(H) and the second intensity threshold I_(L)prior to time p2, no response is provided until delay time p2 haselapsed at time 482. Also in FIG. 4D, dynamic intensity threshold 486decays over time, with the decay starting at time 488 after a predefineddelay time p1 has elapsed from time 482 (when the response associatedwith the second intensity threshold I_(L) was triggered). This type ofdynamic intensity threshold reduces accidental triggering of a responseassociated with the dynamic intensity threshold I_(D) immediately after,or concurrently with, triggering a response associated with a lowerintensity threshold, such as the first intensity threshold I_(H) or thesecond intensity threshold I_(L).

FIG. 4E illustrate yet another dynamic intensity threshold 492 (e.g.,intensity threshold I_(D)). In FIG. 4E, a response associated with theintensity threshold I_(L) is triggered after the delay time p2 haselapsed from when touch input 490 is initially detected. Concurrently,dynamic intensity threshold 492 decays after the predefined delay timep1 has elapsed from when touch input 490 is initially detected. So adecrease in intensity of touch input 490 after triggering the responseassociated with the intensity threshold I_(L), followed by an increasein the intensity of touch input 490, without releasing touch input 490,can trigger a response associated with the intensity threshold I_(D)(e.g., at time 494) even when the intensity of touch input 490 is belowanother intensity threshold, for example, the intensity threshold I_(L).

An increase of characteristic intensity of the contact from an intensitybelow the light press intensity threshold IT_(L) to an intensity betweenthe light press intensity threshold IT_(L) and the deep press intensitythreshold IT_(D) is sometimes referred to as a “light press” input. Anincrease of characteristic intensity of the contact from an intensitybelow the deep press intensity threshold IT_(D) to an intensity abovethe deep press intensity threshold IT_(D) is sometimes referred to as a“deep press” input. An increase of characteristic intensity of thecontact from an intensity below the contact-detection intensitythreshold IT₀ to an intensity between the contact-detection intensitythreshold IT₀ and the light press intensity threshold IT_(L) issometimes referred to as detecting the contact on the touch-surface. Adecrease of characteristic intensity of the contact from an intensityabove the contact-detection intensity threshold IT₀ to an intensitybelow the contact-detection intensity threshold IT₀ is sometimesreferred to as detecting liftoff of the contact from the touch-surface.In some embodiments IT₀ is zero. In some embodiments, IT₀ is greaterthan zero. In some illustrations a shaded circle or oval is used torepresent intensity of a contact on the touch-sensitive surface. In someillustrations, a circle or oval without shading is used represent arespective contact on the touch-sensitive surface without specifying theintensity of the respective contact.

In some embodiments, described herein, one or more operations areperformed in response to detecting a gesture that includes a respectivepress input or in response to detecting the respective press inputperformed with a respective contact (or a plurality of contacts), wherethe respective press input is detected based at least in part ondetecting an increase in intensity of the contact (or plurality ofcontacts) above a press-input intensity threshold. In some embodiments,the respective operation is performed in response to detecting theincrease in intensity of the respective contact above the press-inputintensity threshold (e.g., the respective operation is performed on a“down stroke” of the respective press input). In some embodiments, thepress input includes an increase in intensity of the respective contactabove the press-input intensity threshold and a subsequent decrease inintensity of the contact below the press-input intensity threshold, andthe respective operation is performed in response to detecting thesubsequent decrease in intensity of the respective contact below thepress-input threshold (e.g., the respective operation is performed on an“up stroke” of the respective press input).

In some embodiments, the device employs intensity hysteresis to avoidaccidental inputs sometimes termed “jitter,” where the device defines orselects a hysteresis intensity threshold with a predefined relationshipto the press-input intensity threshold (e.g., the hysteresis intensitythreshold is X intensity units lower than the press-input intensitythreshold or the hysteresis intensity threshold is 75%, 90%, or somereasonable proportion of the press-input intensity threshold). Thus, insome embodiments, the press input includes an increase in intensity ofthe respective contact above the press-input intensity threshold and asubsequent decrease in intensity of the contact below the hysteresisintensity threshold that corresponds to the press-input intensitythreshold, and the respective operation is performed in response todetecting the subsequent decrease in intensity of the respective contactbelow the hysteresis intensity threshold (e.g., the respective operationis performed on an “up stroke” of the respective press input).Similarly, in some embodiments, the press input is detected only whenthe device detects an increase in intensity of the contact from anintensity at or below the hysteresis intensity threshold to an intensityat or above the press-input intensity threshold and, optionally, asubsequent decrease in intensity of the contact to an intensity at orbelow the hysteresis intensity, and the respective operation isperformed in response to detecting the press input (e.g., the increasein intensity of the contact or the decrease in intensity of the contact,depending on the circumstances).

For ease of explanation, the description of operations performed inresponse to a press input associated with a press-input intensitythreshold or in response to a gesture including the press input are,optionally, triggered in response to detecting: an increase in intensityof a contact above the press-input intensity threshold, an increase inintensity of a contact from an intensity below the hysteresis intensitythreshold to an intensity above the press-input intensity threshold, adecrease in intensity of the contact below the press-input intensitythreshold, or a decrease in intensity of the contact below thehysteresis intensity threshold corresponding to the press-inputintensity threshold. Additionally, in examples where an operation isdescribed as being performed in response to detecting a decrease inintensity of a contact below the press-input intensity threshold, theoperation is, optionally, performed in response to detecting a decreasein intensity of the contact below a hysteresis intensity thresholdcorresponding to, and lower than, the press-input intensity threshold.As described above, in some embodiments, the triggering of theseresponses also depends on time-based criteria being met (e.g., a delaytime has elapsed between a first intensity threshold being met and asecond intensity threshold being met).

Although only specific frequencies, amplitudes, and waveforms arerepresented in the sample tactile output patterns in FIG. 4F forillustrative purposes, tactile output patterns with other frequencies,amplitudes, and waveforms may be used for similar purposes. For example,waveforms that have between 0.5 to 4 cycles can be used. Otherfrequencies in the range of 60 Hz-400 Hz may be used as well.

User Interfaces and Associated Processes

Attention is now directed towards embodiments of user interfaces (“UI”)and associated processes that may be implemented on an electronicdevice, such as portable multifunction device 100 or device 300, with adisplay, a touch-sensitive surface, and one or more sensors to detectintensities of contacts with the touch-sensitive surface.

FIGS. 5A-5KKK illustrate example user interfaces and associated tactileoutputs in accordance with some embodiments. The user interfaces inthese figures are used to illustrate the processes described below,including the processes in FIGS. 6A-6D, 8A-8D, 10A-10D, 12A-12D,14A-14B, 16A-16D, 18A-18C, and 20A-20B. For convenience of explanation,some of the embodiments will be discussed with reference to operationsperformed on a device with a touch-sensitive display system 112.However, analogous operations are, optionally, performed on a devicewith a display 450 and a separate touch-sensitive surface 451 inresponse to detecting the contacts on the touch-sensitive surface 451while displaying the user interfaces shown in the figures on the display450, along with a focus selector.

FIGS. 5A-5H illustrate a user interface for displaying control settingsfor tactile outputs and audio outputs, in accordance with someembodiments. For ease of explanation, in the example shown in FIGS.5A-5H, user interface 500 illustrates a tactile output setting and twoaudio output settings in the same user interface. In other embodiments,tactile output settings (e.g., including the tactile output settingshown in user interface 500) may be presented instead in one or moreuser interfaces separate from user interface(s) used for presentingaudio settings (e.g., the audio output settings shown in user interface500). FIGS. 5A-5H illustrate that user interface 500 is displayed ontouch screen 112 of portable multifunction device 100.

FIG. 5A illustrates that user interface 500 includes a slider vibrationcontrol region for controlling a slider vibration setting, indicated bylabel 501 “Slider Vibration” and corresponding switch object 502. Switchobject 502 controls a state of the slider vibration setting. FIG. 5Aillustrates that switch object 502 is initially at the left position(e.g., corresponding to an “off” state of the slider vibration setting).

User interface 500 in FIG. 5A also includes an audio volume controlregion for controlling an audio volume level, indicated by the label“Volume” and corresponding volume slider 504 set to an audio volumelevel corresponding to volume slider position 505. In addition, userinterface 500 includes an audio balance control region for controllingan audio balance setting, indicated by the label “Balance” andcorresponding balance slider 506 set to an audio balance settingcorresponding to balance slider position 507. In some embodiments,device 100 includes one or more audio speakers (e.g., one or morespeakers 111, FIG. 1A). In some embodiments, device 100 generates audiooutputs using the one or more audio speakers included in device 100. Insome embodiments, device 100 transmits audio control signals to one ormore external audio speakers outside of device 100. In some embodiments,the audio volume setting represented by volume slider 504 controls thevolume of audio outputs generated by device 100 (e.g., generated by oneor more audio speakers included in device 100) and/or generated by theone or more external audio speakers based on the audio control signalsfrom device 100. In some embodiments in which device 100 includes aplurality of audio speakers (e.g., a plurality of speakers 111, FIG.1A), the audio balance setting represented by balance slider 506controls the distribution of audio output between the plurality of audiospeakers. In some embodiments in which device 100 transmits audiocontrol signals to a plurality of external audio speakers, the audiobalance setting represented by balance slider 506 controls thedistribution of audio output between the plurality of external audiospeakers. For example, as shown in FIG. 5A, the audio balance settingrepresented by balance slider 506 controls the distribution of audiooutput between a speaker labeled “L” (e.g., a left audio channelspeaker) and a speaker labeled “R” (e.g., a right audio channelspeaker).

FIG. 5B illustrates a transition of user interface 500 from userinterface 500 in FIG. 5A. In particular, FIG. 5B illustrates user input503 (e.g., a tap gesture) detected while displaying user interface 500as presented in FIG. 5A. In response to detecting user input 503, switchobject 502 is toggled from the left position to the right position(e.g., in accordance with user input 503 toggling switch object 502, theslider vibration setting is toggled from the “off state” to an “on”state). Also, in response to detecting user input 503, device 100generates tactile output 508 (e.g., MicroTap Medium (150 Hz), gain max:0.8, gain min: 0.0).

FIGS. 5C-5E illustrate a series of transitions of user interface 500from user interface 500 in FIG. 5B. In particular, FIGS. 5C-5Eillustrate generation of tactile outputs in response to changes involume slider position 505 with respect to volume slider 504. In someembodiments, as shown in FIGS. 5C-5E, tactile outputs are generatedusing a tactile output level (e.g., gain) that corresponds to therespective audio volume level indicated by a respective volume sliderposition.

For example, FIG. 5C illustrates movement of user input 509 toward theleft edge of device 100 (e.g., user input 509 is an input gesture, suchas a drag gesture, leftward from an initial contact at a positioncorresponding to volume slider position 505, as illustrated in FIG. 5B).In response to user input 509, the audio volume level indicated byvolume slider 504 is changed to an audio volume level corresponding tovolume slider position 505-b (e.g., from an audio volume levelcorresponding to volume slider position 505, FIG. 5B). In addition,device 100 generates tactile output 510. In accordance with volumeslider position 505-b indicating an audio volume level near a minimumaudio volume level (e.g., zero audio volume), tactile output 510 isgenerated as a light tactile output (e.g., MicroTap Medium (150 Hz) witha gain of 0.1).

FIG. 5D illustrates a transition of user interface 500 from userinterface 500 in FIG. 5C. In particular, FIG. 5D illustrates movement ofuser input 509 away from the left edge of device 100. In response touser input 509, the audio volume level indicated by volume slider 504 ischanged to an audio volume level corresponding to volume slider position505-c (e.g., from an audio volume level corresponding to volume sliderposition 505-b, FIG. 5C). In addition, device 100 generates tactileoutput 511. In accordance with volume slider position 505-c indicatingan audio volume level near a moderate audio volume level, tactile output511 is generated as a moderate tactile output (e.g., MicroTap Medium(150 Hz) with a gain of 0.4).

FIG. 5E illustrates a transition of user interface 500 from userinterface 500 in FIG. 5D. In particular, FIG. 5E illustrates movement ofuser input 509 toward the right edge of device 100. In response to userinput 509, the audio volume level indicated by volume slider 504 ischanged to an audio volume level corresponding to volume slider position505-d (e.g., from an audio volume level corresponding to volume sliderposition 505-c, FIG. 5D). In addition, device 100 generates tactileoutput 512. In accordance with volume slider position 505-d indicatingan audio volume level near a maximum audio volume level, tactile output512 is generated as a strong tactile output (e.g., MicroTap Medium (150Hz) with a gain of 0.7).

FIG. 5F illustrates a transition of user interface 500 from userinterface 500 in FIG. 5E. In particular, FIG. 5F illustrates continuedmovement of user input 509 toward the right edge of device 100. Inresponse to user input 509, the audio volume level indicated by volumeslider 504 is changed to an audio volume level corresponding to volumeslider position 505-e (e.g., from an audio volume level corresponding tovolume slider position 505-d, FIG. 5E). In the example shown in FIG. 5F,volume slider position 505-e is an end position of volume slider 504(e.g., corresponding to a maximum audio volume level). Accordingly,device 100 generates tactile output 513 (e.g., MicroTap Medium (150 Hz),gain max: 0.6, gain min: 0.3) to indicate that an end of volume slider504 has been reached.

FIGS. 5G-5H illustrate a series of transitions of user interface 500from user interface 500 in FIG. 5F. In particular, FIGS. 5G-5Hillustrate generation of tactile outputs in response to changes inbalance slider position 507 with respect to balance slider 506. Asdiscussed above with respect to FIG. 5A, the audio balance settingrepresented by balance slider 506 controls the distribution of audiooutput between a speaker labeled “L” (e.g., a left audio channelspeaker) and a speaker labeled “R” (e.g., a right audio channelspeaker). In some embodiments, as shown in FIGS. 5G-5H, tactile outputsare generated using a tactile output level (e.g., gain) that correspondsto the respective audio balance setting indicated by a respectivebalance slider position.

For example, FIG. 5G illustrates movement of user input 514 toward theleft edge of device 100 (e.g., user input 514 is an input gesture, suchas a drag gesture, leftward from an initial contact at a positioncorresponding to balance slider position 507, as illustrated in FIG.5F). In response to user input 514, the audio balance setting indicatedby balance slider 506 is changed (e.g., from an audio balance settingcorresponding to balance slider position 507, FIG. 5F) to an audiobalance setting corresponding to balance slider position 507-b. As shownin FIG. 5G, balance slider position 507-b is near an edge (e.g., theleft edge) of balance slider 506. In some embodiments, balance sliderposition 507-b indicates an audio balance setting having close to amaximum allocation of audio output to the “L” speaker and close to aminimum allocation of audio output to the “R” speaker. In response touser input 514, device 100 generates left-side tactile output 515-a(e.g., using one or more tactile output generator(s) 167, FIG. 1A, thatare located in a left region of device 100) and right-side tactileoutput 515-b (e.g., using one or more tactile output generator(s) 167,FIG. 1A, that are located in a right region of device 100). In addition,in accordance with balance slider position 507-b indicating an audiobalance setting having close to a maximum allocation of audio output tothe “L” speaker, left-side tactile output 515-a is generated as a strongtactile output (e.g., MicroTap Medium (150 Hz) with a gain of 0.7). Inaccordance with balance slider position 507-b indicating an audiobalance setting having close to a minimum allocation of audio output tothe “R” speaker, right-side tactile output 515-b is generated as a lighttactile output (e.g., MicroTap Medium (150 Hz) with a gain of 0.1).

FIG. 5H illustrates a transition of user interface 500 from userinterface 500 in FIG. 5G. In particular, FIG. 5H illustrates movement ofuser input 514 away from the left edge and toward the right edge ofdevice 100. In response to user input 514, the audio balance settingindicated by balance slider 506 is changed (e.g., from an audio balancesetting corresponding to balance slider position 507-b, FIG. 5G) to anaudio balance setting corresponding to balance slider position 507-c. Asshown in FIG. 5H, balance slider position 507-c is near an edge (e.g.,the right edge) of balance slider 506. In some embodiments, balanceslider position 507-c indicates an audio balance setting having close toa maximum allocation of audio output to the “R” speaker and close to aminimum allocation of audio output to the “L” speaker. In response touser input 514, device 100 generates left-side tactile output 516-a(e.g., using one or more tactile output generator(s) 167, FIG. 1A, thatare located in a left region of device 100) and right-side tactileoutput 516-b (e.g., using one or more tactile output generator(s) 167,FIG. 1A, that are located in a right region of device 100). In addition,in accordance with balance slider position 507-c indicating an audiobalance setting having close to a maximum allocation of audio output tothe “R” speaker, right-side tactile output 516-b is generated as astrong tactile output (e.g., MicroTap Medium (150 Hz) with a gain of0.7). In accordance with balance slider position 507-c indicating anaudio balance setting having close to a minimum allocation of audiooutput to the “L” speaker, left-side tactile output 516-a is generatedas a light tactile output (e.g., MicroTap Medium (150 Hz) with a gain of0.1).

FIGS. 5I-5J illustrate an example user interface for generating tactileoutputs when a user input moves across displayed menu items, inaccordance with some embodiments. In particular, FIG. 5I illustratesuser input 521 (e.g., a contact) detected at a location on touch screen112 that corresponds to keyboard interface menu icon 522 in userinterface 520. In response to detecting user input 521, keyboardinterface menu 523 is displayed. Keyboard interface menu 523 includes aplurality of displayed menu item affordances representing keyboardinterface options, including keyboard option affordance 524 (forselecting an English-language keyboard interface), keyboard optionaffordance 525 (for selecting an Emoji keyboard interface), and keyboardoption affordance 526 (for selecting a French-language keyboardinterface). Prior to movement of user interface 521, a current selectionindication (e.g., shading) is displayed on keyboard option affordance524 to indicate that the keyboard interface corresponding to keyboardoption affordance 524 (e.g., an English-language keyboard interface) isthe currently displayed keyboard interface.

FIG. 5J illustrates a series of transitions of user interface 520 overtime from 520 in FIG. 5I. FIG. 5J illustrates movement of user input 521(e.g., user input 521 is a continuous input gesture from an initialcontact at a position corresponding to keyboard option menu icon 522, asillustrated in FIG. 5I) across the displayed menu item affordances ofkeyboard option menu 523. At T=t₁, user input 521 has moved towardkeyboard option affordance 526 but has not reached a position thatcorresponds to (e.g., activates selection of) keyboard option affordance526. Accordingly, the current selection indication continues to bedisplayed on keyboard option affordance 526.

At T=t₂, user input 521 has moved to a position that activates selectionof keyboard option affordance 526. In accordance with user input 521activating selection of keyboard option affordance 526, tactile output528 is generated (e.g., MicroTap High (270 Hz), gain: 0.4). In addition,a current selection indication (e.g., shading) is displayed on keyboardoption affordance 526 to indicate that keyboard option affordance 526 isthe currently selected keyboard option affordance. Accordingly, thecurrent selection indication is no longer displayed on keyboard optionaffordance 524. In some embodiments, upon ceasing to detect user input521 while the current selection indication is displayed on keyboardoption affordance 526, the keyboard interface corresponding to keyboardoption affordance 526 becomes the currently displayed keyboardinterface.

At T=t₃, user input 521 has moved toward keyboard option 525 but has notreached a position that corresponds to (e.g., activates selection of)keyboard option affordance 525. Accordingly, the current selectionindication continues to be displayed on keyboard option affordance 526.

At T=t₄, user input 521 has moved to a position that activates selectionof keyboard option affordance 525. In accordance with user input 521activating selection of keyboard option affordance 525, tactile output529 is generated. In some embodiments, a tactile output is generated inaccordance with a user input activating selection of a respective menuitem affordance (e.g., one of keyboard option affordances 524, 525, 526)in a menu (e.g., keyboard interface menu 523). In some embodiments, atactile output is generated in accordance with a user input activatingselection of a respective menu item affordance that is distinct from apreviously selected menu item. In some embodiments, tactile outputsgenerated in accordance with a user input activating selection ofrespective menu item affordances in keyboard interface menu 523 are of asame type. For example, tactile output 529, generated at T=t₄ inaccordance with user input 521 activating selection of keyboard optionaffordance 525, is a same type of tactile output (e.g., MicroTap High(270 Hz), gain: 0.4) as tactile output 528, generated at T=t₃ inaccordance with user input 521 activating selection of keyboard optionaffordance 526.

FIGS. 5K-5O illustrate an example user interface for generating tactileoutputs when a movable component rotates in accordance with one or morescroll inputs, in accordance with some embodiments. In the example shownin FIGS. 5K-5O, user interface 530 displayed on touch screen 112includes a user interface for a time picker (e.g., generated by using aUIDatePicker class described in Appendix B). User interface 530 includesmovable component 531 that further includes movable elements, e.g., hourwheel 532 for selecting an hour value from 0-23 hours and minute wheel533 for selecting a minute value from 0-59 min. In some embodiments,user interface 530 is a date picker (e.g., including a movable componentfor selecting a year, a month, and a date value from a plurality ofyear, month, and date values, respectively). In some embodiments,tactile outputs are generated when movable component 531, or a movableelement of movable component 531, such as hour wheel 532 or minute wheel533, moves through respective markers (e.g., a position marked by astationary indicator, such as “hours” for hour wheel 532, or “min” forminute wheel 533).

FIGS. 5K-5O illustrate moving minute wheel 533 and tactile outputsgenerated in connection with moving minute wheel 533. FIG. 5Killustrates that minute wheel 533 of movable component 531 initiallydisplays a selected minute value of “59 min.” In addition, FIG. 5Killustrates user input 534 (e.g., a scroll input) detected by device 100at a location on touch screen 112 corresponding to minute wheel 533. Insome embodiments, user input 534 includes downward movement of a contactfrom an initial position corresponding to minute wheel 533 andsubsequent liftoff of the contact.

FIGS. 5L-5O illustrate that, in response to detecting user input 534,minute wheel 533 is moved through respective markers. In someembodiments, as illustrated in FIGS. 5L-5O, after device 100 ceases todetect user input 534 (e.g., detecting liftoff of the contact of userinput 534), minute wheel 533 continues to move (e.g., rotate) inaccordance with a moment of inertia, and the continued movement slowsdown gradually until minute wheel 533 stops.

FIG. 5L illustrates that in response to user input 534, minute wheel 533has moved to a minute value “58 min.” In accordance with minute wheel533 moving to minute value “58 min,” device 100 generates tactile output535 (e.g., MicroTap High (270 Hz) with a gain of 0.4 and minimuminterval 0.05 seconds). In some embodiments, device 100 also generatesan audio output (e.g., an audio output that accompanies the tactileoutput).

FIG. 5M illustrates that minute wheel 533 has moved to a minute value“56 min” (e.g., after moving through a minute value “57 min”). In someembodiments, generation of a tactile output corresponding to minutewheel 533 moving to a respective marker is forgone when minute wheel 533moves fast in response to a user input (e.g., in response to a flickgesture). For example, a time interval is determined from a point intime corresponding to a most recently generated tactile output and apoint in time corresponding to the anticipated tactile outputcorresponding to minute wheel 533 moving to the respective marker. Insome embodiments, in accordance with a determination that the timeinterval is less than a threshold value (e.g., 0.05 seconds), generationof the tactile output corresponding to minute wheel 533 moving to therespective marker is forgone. For example, as illustrated in FIG. 5M, notactile output is generated when minute wheel 533 moves to “56 min”because the time interval between a time corresponding to generation ofa most recently generated tactile output (e.g., tactile output 535corresponding to minute wheel 533 passing “58 min,” FIG. 5L) and a timecorresponding to anticipated generation of a tactile outputcorresponding to minute wheel 533 passing “56 min” is less than athreshold value. In other words, a tactile output is skipped when atactile output rate limit is reached, e.g., the determined time intervalis less than a minimum interval threshold (e.g., 0.05 seconds).

FIG. 5N illustrates that minute wheel 533 has moved to a minute value“54 min” (e.g., after moving through a minute value “55 min”). Asillustrated in FIG. 5N, device 100 generates tactile output 536 (e.g.,MicroTap High (270 Hz) with a gain of 0.4 and minimum interval 0.05seconds) in conjunction with minute wheel 533 moving to minute value “54min.” In some embodiments, a time interval is determined between a timecorresponding to the most recently generated tactile output (e.g.,tactile output 535 corresponding to minute wheel 533 passing “58 min,”FIG. 5L) and a time corresponding to tactile output 536. In someembodiments, tactile output 536 is generated in accordance with thedetermined time interval being at least (e.g., greater than or equal to)the threshold value.

FIG. 5O illustrates that minute wheel 533 has moved to a minute value“53 min” (e.g., from minute value “54 min,” FIG. 5N). As illustrated inFIG. 5O, no tactile output is generated when minute wheel 533 moves to“53 min” because a time interval between a time corresponding togeneration of a most recently generated tactile output (e.g., tactileoutput 536 corresponding to minute wheel 533 passing “54 min,” FIG. 5N)and a time corresponding to anticipated generation of a tactile outputcorresponding to minute wheel 533 passing “53 min” is less than thethreshold value.

FIGS. 5P-5II illustrate example user interfaces for generating tactileoutputs to indicate thresholds associated with user interface elements.

FIGS. 5P-5W illustrate an example user interface for generating atactile outputs to indicate a threshold associated with a list of e-mailsummary items in an e-mail application (e.g., e-mail client module 140,FIGS. 1A and 3), such as a threshold for refreshing content displayed inthe e-mail application. In some embodiments, the example user interfaceis generated using a UIRefreshControl class described in Appendix B. Inthe example shown in FIGS. 5P-5W, user interface 540 includes a list ofe-mail summary items that includes e-mail summary items 541, 542, and543. A dashed line (which is typically not displayed to a user)represents edge 549 (e.g., a top edge) of the list of e-mail summaryitems. A current status of the list of e-mail summary items (e.g.,“Updated Just Now”) is indicated by status indicator field 547. A seconddashed line labeled D_(th) indicates a movement threshold for refreshingcontent displayed in the list of e-mail summary items of user interface540.

In FIG. 5P, device 100 detects user input 544 (e.g., a contact) at alocation on touch screen 112 that corresponds to the list of e-mailsummary items (e.g., corresponding to e-mail summary item 541).

FIG. 5Q illustrates a transition of user interface 540 from userinterface 540 in FIG. 5P. In particular, FIG. 5Q illustrates downwardmovement of user input 544 (e.g., user input is a drag gesture thatincludes an initial contact on touch screen 112 as illustrated in FIG.5P followed by downward movement of the contact.) In response to thedownward movement of user input 544, the list of e-mail summary itemsmoves downward (e.g., e-mail summary items 541, 542, and 543 movedownward in user interface 540), and, accordingly, edge 549 of the listof e-mail summary items moves downward. In some embodiments, in responseto user input 544, a progress indicator 548 is displayed in a regionabove the list of e-mail summary items revealed by the downward movementof the list of e-mail summary items. Progress indicator 548 indicates,for example, whether the movement of user input 544 meets refreshcriteria for refreshing the list of e-mail summary items and/or whethera refresh process to download and present newly received e-mails isongoing. As shown in FIG. 5Q, prior to user input 544 crossing movementthreshold D_(th), progress indicator 548 is displayed with less than afull ring of progress indicator spokes.

FIG. 5R illustrates a transition of user interface 540 from userinterface 540 in FIG. 5Q. In particular, FIG. 5R illustrates furtherdownward movement of user input 544 from its position as shown in FIG.5Q (e.g., a continuation of the drag gesture shown in FIG. 5Q). Inresponse to the downward movement of user input 544, the list of e-mailsummary items moves further downward from its position as shown in FIG.5Q (e.g., e-mail summary items 541, 542, and 543 move further downwardin user interface 540). As shown in FIG. 5R, in response to user input544, edge 549 of the list of e-mail summary items has crossed movementthreshold D_(th). Accordingly, in some embodiments, progress indicator548 is displayed with a full ring of progress indicator spokes. In someembodiments, the status indicated in status indicator field 547 isupdated to indicate that a content refresh process is initiated (e.g.,“Checking for Mail . . . ”), such as a content refresh process to checkfor e-mail received since a last time that content was refreshed. Inaddition, device 100 generates tactile output 545. In some embodiments,device 100 generates tactile output 545 (e.g., MicroTap High (270 Hz),gain: 0.6) in response to edge 549 crossing movement threshold D_(th).In some embodiments, in accordance with continued movement of user input544, edge 549 is at a position beyond movement threshold D_(th) whentactile output 545 is generated.

FIG. 5S illustrates a transition of user interface 540 from userinterface 540 in FIG. 5R. In particular, FIG. 5S illustrates userinterface 540 in response to ceasing to detect user input 544 (e.g., inresponse to liftoff of the contact by user input 544) from a positioncorresponding to edge 549 crossing movement threshold D_(th) (e.g., asdescribed above with reference to FIG. 5R). In response to ceasing todetect user input 544, the list of e-mail summary items is released andgradually returns to its initial position (e.g., moves upward), asillustrated in FIGS. 5S-5T. In addition, status indicator field 547 isupdated to indicate that a new e-mail message is being downloaded (e.g.,“Downloading 1 of 1”). In the example illustrated in FIG. 5S, progressindicator 548 is updated to indicate that a content refresh process isin progress.

FIG. 5T illustrates a transition of user interface 540 from userinterface 540 in FIG. 5S. In particular, FIG. 5S illustrates that, afterdevice 100 ceases to detect user input 544, the list of e-mail summaryitems has returned to its initial position as illustrated in FIG. 5P. Inaccordance with the content refresh process described above withreference to FIGS. 5R-5S, an e-mail summary item 546 that corresponds toa new downloaded e-mail message is shown in the list of e-mail summaryitems. In some embodiments, as illustrated in FIG. 5T, status indicatorfield 547 is updated to indicate that the list of e-mail summary itemshas been updated (e.g., “Updated Just Now”).

FIGS. 5U-5V are similar to FIGS. 5P-5Q, as described above. Inparticular, FIG. 5U illustrates user input 544 (e.g., an initialcontact) detected at a location on touch screen 112 corresponding to thelist of e-mail summary items (e.g., corresponding to e-mail summary item541) in user interface 540, and FIG. 5V illustrates a transition of userinterface 540 in response to downward movement of user input 544 (e.g.,a drag gesture that includes the initial contact followed by downwardmovement of the contact), including downward movement of the list ofe-mail summary items and edge 549. In some embodiments, if device 100ceases to detect user input 544 (e.g., device 100 detects liftoff of thecontact) before edge 549 has crossed movement threshold D_(th), atactile output is not generated in response to movement of edge 549.FIG. 5W illustrates an example of such a transition of user interface540 from user interface 540 in FIG. 5V. FIG. 5W illustrates that, inresponse to ceasing to detect user input 544, user interface 540 returnsto its initial position (e.g., moves upward), and a tactile output isnot generated.

FIGS. 5X-5Y illustrate an example user interface for generating atactile output to indicate a threshold associated with a list of e-mailsummary items in an e-mail application (e.g., e-mail client module 140,FIGS. 1A and 3), such as a threshold at an edge of the list of e-mailsummary items. In the example shown in FIGS. 5X-5Y, user interface 550includes a list of e-mail summary items that includes e-mail summaryitems 551, 552, 553, 554, 555, and 556. A dashed line (which istypically not displayed to a user) represents a threshold 557, whichindicates a bottom edge of a display region for the list of e-mailsummary items. As illustrated in FIG. 5X, at least a portion of e-mailsummary item 556 in the list of e-mail summary items is below threshold557. Accordingly, the portion of e-mail summary item 556 that is belowthreshold 557 is not displayed. Additionally, FIG. 5X illustrates a userinput 558 detected at a location on touch screen 112 that corresponds tothe list of e-mail summary items. For example, as shown in FIG. 5X, userinput 558 includes an initial contact that corresponds to e-mail summaryitem 554 of the list of e-mail summary items followed by upward movementof the contact.

FIG. 5Y illustrates that, in response to the upward movement of userinput 558, the list of e-mail summary items moves upward (e.g., e-mailsummary items 551, 552, 553, 554, and 556 move upward in user interface550), such that e-mail summary item 556 is displayed entirely in thedisplay region for the list of e-mail summary items. Accordingly, aportion of e-mail summary item 551 (e.g., the topmost e-mail summaryitem) moves past a top edge of the display region for the list of e-mailsummary items. As shown in FIG. 5Y, the portion of e-mail summary itemabove the top edge of the display region is not displayed. In accordancewith the upward movement of the list of e-mail summary items such thate-mail summary item 556 is displayed entirely, bottom edge 556-b ofe-mail summary item 556, which is represented by a second, thickerdashed line, crosses threshold 557. In some embodiments, in accordancewith bottom edge 556-b crossing threshold 557, device 100 generatestactile output 559 (e.g., MicroTap High (270 Hz), gain: 0.6).

FIGS. 5Z-5II illustrate an example user interface for generating tactileoutputs to indicate thresholds associated with zoom operations. Inparticular, FIGS. 5Z-5EE illustrate an example user interface forgenerating a tactile output in accordance with a zoom scale of displayeduser interface elements crossing a maximum zoom threshold during azoom-in operation. FIGS. 5FF-5II illustrate an example user interfacefor generating a tactile output in accordance with a zoom scale ofdisplayed user interface elements crossing a minimum zoom thresholdduring a zoom-out operation. In the example shown in FIGS. 5Z-5II, userinterface 560 includes a user interface of a web browser application(e.g., browser module 147, FIG. 1A).

FIG. 5Z illustrates user interface 560 displayed on touch screen 112. Insome embodiments, as shown in FIG. 5Z, user interface 560 includes userinterface region 561, indicated by the dashed outline (which istypically not displayed to a user). For example, in FIG. 5Z, userinterface region 561 includes content of a webpage that corresponds tothe web address “apple.com,” including a plurality of user interfaceelements (e.g., a top menu bar including an apple icon, icons for“MacBook Air,” “MacBook Pro,” and “iMac,” etc.). Degree of zoom scale563 indicates a zoom scale of user interface region 561 with respect tominimum zoom threshold Z_(min) and maximum zoom threshold Z_(max). InFIG. 5Z, for example, degree of zoom scale 563 indicates that the zoomscale of user interface region 561 corresponds to minimum zoom thresholdZ_(min). FIG. 5Z also illustrates user input 562 corresponding to adepinch gesture (e.g., detection of two finger contacts at the initialpositions shown in FIG. 5Z and movement of the two contacts away fromeach other as indicated by the arrows) detected on user interface 560 atlocations corresponding to (e.g., within) user interface region 561. Insome embodiments, the user interface illustrated in FIG. 5Z is generatedby using a UIZoomEdgeFeedbackGenerator class described in Appendix B.

FIG. 5AA illustrates a transition of user interface 560 from userinterface 560 in FIG. 5Z. In some embodiments, in response to detectinguser input 562 (e.g., the depinch gesture), user interface region 561 isexpanded (e.g., the zoom scale of user interface region 561 isincreased) such that user interface region 561 extends beyond touchscreen 112 (e.g., an outer edge of user interface region 561 extendsbeyond one or more outer edges of touch screen 112). As a result, only aportion of user interface region 561 is displayed on touch screen 112(e.g., portions of the top menu bar, including the apple icon, andportions of the icons for “MacBook Air” and “iMac” are not displayed ontouch screen 112). Degree of zoom scale 563 indicates that the zoomscale of user interface region 561 has increased above minimum zoomthreshold Z_(min).

FIG. 5BB illustrates a transition of user interface 560 from userinterface 560 in FIG. 5AA. In particular, FIG. 5BB illustrates that, inresponse to a continuation of user input 562 (e.g., a continuation ofthe depinch gesture by continued movement of the two contacts away fromeach other), user interface region 561 is further expanded (e.g., thezoom scale of user interface region 561 is further increased) such thatuser interface region 561 extends further beyond touch screen 112. As aresult, a lesser portion of user interface 561 is displayed on touchscreen 112 in FIG. 5BB than in FIG. 5AA (e.g., the top menu bar and theicons for “MacBook Air,” “MacBook Pro,” and “iMac” are not displayed ontouch screen 112). Degree of zoom scale 563 indicates that the zoomscale of user interface region 561 has increased further above minimumzoom threshold Z_(min) but remains below maximum zoom threshold Z_(max).

FIG. 5CC illustrates a transition of user interface 560 from userinterface 560 in FIG. 5BB. In particular, FIG. 5CC illustrates that, inresponse to a further continuation of user input 562 (e.g., a furthercontinuation of the depinch gesture by further continued movement of thetwo contacts away from each other), user interface region 561 is furtherexpanded (e.g., the zoom scale of user interface region 561 is furtherincreased) such that user interface region 561 extends further beyondtouch screen 112. As a result, a lesser portion of user interface 561 isdisplayed on touch screen 112 in FIG. 5CC than in FIG. 5BB. Degree ofzoom scale 563 indicates that the zoom scale of user interface region561 has increased further above minimum zoom threshold Z_(min) tomaximum zoom threshold Z_(max). In accordance with the zoom scale ofuser interface region 561 reaching maximum zoom threshold Z_(max),device 100 generates tactile output 564 (e.g., MicroTap High (270 Hz),gain: 0.6).

FIG. 5DD illustrates a transition of user interface 560 from userinterface 560 in FIG. 5CC. In particular, FIG. 5DD illustrates that, inresponse to a continuation of user input 562 (e.g., the two contacts ofthe depinch gesture continue to be detected after further movement ofthe two contacts away from each other), user interface region 561 isfurther expanded (e.g., the zoom scale of user interface region 561 isfurther increased) such that user interface region 561 extends furtherbeyond touch screen 112. As a result, a lesser portion of user interface561 is displayed on touch screen 112 in FIG. 5DD than in FIG. 5CC.Degree of zoom scale 563 indicates that the zoom scale of user interfaceregion 561 has increased above maximum zoom threshold Z_(max). In someembodiments, the zoom scale of user interface region 561 is increasedtemporarily above maximum zoom threshold Z_(max) in response to thecontinuation of user input 562.

FIG. 5EE illustrates a transition of user interface 560 from userinterface 560 in FIG. 5DD. In particular, FIG. 5EE illustrates userinterface 560 in response to ceasing to detect user input 562 (e.g., inresponse to liftoff of the two contacts of the depinch gesture). WhileFIG. 5DD illustrates that the zoom scale of user interface region 561 istemporarily increased above maximum zoom threshold Z_(max) in responseto user input 562, FIG. 5EE illustrates that, in response to ceasing todetect user input 562, the zoom scale of user interface region 561 isrestored to Z_(max), as indicated by degree of zoom scale 563.

FIGS. 5FF-5II illustrate alternate transitions of user interface 560corresponding to generating a tactile output in accordance with the zoomscale of user interface region 561 crossing minimum zoom thresholdZ_(min) during a zoom-out operation. FIG. 5FF is similar to FIG. 5AA andillustrates that user interface region 561 of user interface 560 isexpanded such that user interface region 561 extends beyond touch screen112 (e.g., user interface region 561 extends beyond one or more outeredges of touch screen 112). As a result, only a portion of userinterface region 561 is displayed on touch screen 112 (e.g., portions ofthe top menu bar, including the apple icon, and portions of the iconsfor “MacBook Air” and “iMac” are not displayed on touch screen 112).Degree of zoom scale 563 indicates that the zoom scale of user interfaceregion 561 is above minimum zoom threshold Z_(min) at a zoom scalesimilar to the zoom scale shown in FIG. 5AA. FIG. 5FF also illustratesuser input 565 corresponding to a pinch gesture (e.g., detection of twofinger contacts at the initial positions shown in FIG. 5FF and movementof the two contacts toward each other as indicated by the arrows)detected on user interface 560 at locations corresponding to (e.g.,within) user interface region 561.

FIG. 5GG illustrates a transition of user interface 560 from userinterface 560 in FIG. 5FF. In some embodiments, in response to detectinguser input 565 (e.g., the pinch gesture), device 100 shrinks userinterface region 561 (e.g., decreases the zoom scale of user interfaceregion 561) such that user interface region 561 does not extend beyondtouch screen 112 (e.g., user interface region 561 is fully displayed ontouch screen 112). Degree of zoom scale 563 indicates that the zoomscale of user interface region 561 is at minimum zoom threshold Z_(min)(e.g., the zoom scale of user interface region 561 has decreased fromthe zoom scale shown in FIG. 5FF to Z_(min)). In response to the zoomscale of user interface region 561 reaching minimum zoom thresholdZ_(min), device 100 generates tactile output 566 (e.g., MicroTap High(270 Hz), gain: 0.6).

FIG. 5HH illustrates a transition of user interface 560 from userinterface 560 in FIG. 5GG. In FIG. 5HH, in response to detecting acontinuation of user input 565 (e.g., the two contacts of the pinchgesture continue to be detected after further movement of the twocontacts toward each other), device 100 shrinks user interface further(e.g., the zoom scale of user interface region 561 is further decreased)such that an outer edge of user interface region 561 is within touchscreen 112. Degree of zoom scale 563 indicates that the zoom scale ofuser interface region 561 has decreased below minimum zoom thresholdZ_(min). In some embodiments, the zoom scale of user interface region561 is decreased temporarily below minimum zoom threshold Z_(min) inresponse to the continuation of user input 565.

FIG. 5II illustrates a transition of user interface 560 from userinterface 560 in FIG. 5HH. In particular, FIG. 5II illustrates userinterface 560 in response to ceasing to detect user input 565 (e.g., inresponse to liftoff of the two contacts of the pinch gesture). WhileFIG. 5HH illustrates that the zoom scale of user interface region 561 istemporarily decreased below minimum zoom threshold Z_(min) in responseto user input 565, FIG. 5II illustrates that, in response to ceasing todetect user input 565, the zoom scale of user interface region 561 isrestored to Z_(min), as indicated by degree of zoom scale 563.

FIGS. 5JJ-5SS illustrate example user interfaces for generating tactileoutputs with respect to operations for picking up, dragging, anddropping user interface objects, in accordance with some embodiments.

FIGS. 5JJ-5KK illustrate an example list user interface for generatingtactile outputs with respect to rearranging list elements in the list,in accordance with some embodiments. The rearrangement is performed dueto movement of a list element in accordance with movement of a userinput (e.g., a drag gesture that includes an initial contact followed bymovement of the contact). In FIG. 5JJ, user interface 570 a includes anexample clock interface for editing a list of clocks for displayinglocal times associated with a plurality of cities (e.g., based on timezones associated with the cities “Cupertino,” “New York,” “Tokyo,” and“Melbourne”). User interface 570 a includes a plurality of predeterminedsnap positions that correspond to a plurality of positions fordisplaying list elements. As shown in FIG. 5JJ, the list elements (e.g.,clocks) are displayed in respective predetermined snap positions,including list element 572, labeled “Tokyo,” and list element 573,labeled “New York.” FIG. 5JJ also illustrates a user input 571 detectedat a location on touch screen 112 that corresponds to list element 572(e.g., corresponding to a reordering affordance located near the rightedge of list element 572).

FIG. 5KK illustrates a series of transitions of user interface 570 afrom user interface 570 a in FIG. 5JJ. The series of transitions of userinterface 570 a includes user interfaces 570 b, 570 c, 570 d, and 570 ecorresponding to first, second, third, and fourth portions of user input571, respectively. In FIG. 5KK, time T=t₀ corresponds to user interface570 a as shown in and described above with reference to FIG. 5JJ. Attime T=t₁, user interface 570 b illustrates that list element 572 isselected in response to the first portion of user input 571 (e.g., aninitial contact or a long press gesture). Device 100 visually indicatesthat list element 572 is selected (e.g., by highlighting and/orenlarging selected list element 572). Optionally, in some embodiments,in response to the first portion of user input 571 selecting listelement 572, device 100 generates tactile output 574 b (e.g., MicroTapHigh (270 Hz) with a gain of 1.0) to indicate that list element 572 hasbeen selected.

In some embodiments, while list element 572 is selected, the secondportion of user input 571 is detected. For example, the second portionof user input 571 includes continued contact followed by upward movementof the contact (e.g., upward movement following the initial contact orlong press gesture). At time T=t₂, user interface 570 c illustratesthat, in accordance with upward movement of user input 571 during thesecond portion of user input 571, selected list element 572 is movedupwards. For example, selected list element 572 moves upward towards aposition of unselected list element 573 (e.g., “New York”). Accordingly,selected list element 572 is displayed over at least a portion ofunselected list element 573.

At time T=t₃, user interface 570 d illustrates continued upward movementof user input 571, detected during the third portion of user input 571,from the position of user input 571 as shown in user interface 570 ccorresponding to time T=t₂. In accordance with the continued upwardmovement of user input 571, selected list element 572 moves into apredetermined snap position corresponding to the previous position ofunselected list element 573 (e.g., as shown in user interface 570 c). Insome embodiments, the predetermined snap position corresponding to theprevious position of selected list element 572 (e.g., as shown in userinterface 570 a) becomes vacant. In accordance with selected listelement 572 moving into the predetermined snap position corresponding tothe previous position of unselected list element 573, unselected listelement 573 moves downward into the predetermined snap positioncorresponding to the previous position of selected list element 572(e.g., unselected list element 573 and selected list element 572 aredisplayed as having switched positions). In some embodiments, inconjunction with selected list element 572 moving into the predeterminedsnap position previously corresponding to unselected list element 573,device 100 generates tactile output 574 d (e.g., MicroTap High (270 Hz)with a gain of 1.0).

At time T=t₄, device 100 has ceased to detect user input 571 (e.g.,device 100 has detected the fourth portion of user input 571, the fourthportion of user input 571 including liftoff of the contact) whileselected list element 572 is at the predetermined snap position as shownin user interface 570 d corresponding to time T=t₃. Accordingly, listelement 572 settles into and is displayed (e.g., as an unselected listelement) in the position previously occupied by list element 573, whilelist element 573 is displayed in the position previously occupied bylist element 572 (e.g., as shown in user interface 570 a). Optionally,in some embodiments, in response to list element 572 settling into thelist of clocks, device 100 generates tactile output 574 e (e.g.,MicroTap High (270 Hz) with a gain of 0.6) to indicate that list element572 has been dropped.

FIGS. 5LL-5QQ illustrate an example calendar user interface forgenerating tactile outputs with respect to moving calendar objects, inaccordance with some embodiments. FIG. 5LL illustrates that userinterface 575 includes two existing calendar objects corresponding toexisting calendar entries (e.g., “Have Lunch” and “Go to Gym”). Calendarobject 576, labeled “Go to Gym” is initially scheduled for Wednesday,September 3 at 11:15 AM. In some embodiments, user interface 575 isgenerated by using a UITableView class described in Appendix B.

FIG. 5MM illustrates a transition of user interface 575 from userinterface 575 in FIG. 5LL. In particular, FIG. 5MM illustrates a firstportion of user input 577 (e.g., a long press gesture) detected at alocation on touch screen 112 that corresponds to calendar object 576. Inresponse to the first portion of user input 577, calendar object 576 isselected (e.g., picked up). In some embodiments, a visual indication isdisplayed to indicate that a calendar object has been selected, suchthat the calendar object appears to be lifted from the surface of a userinterface in the virtual z-direction and/or floating on or above thesurface of the user interface. For example, in FIG. 5MM, selectedcalendar object 576 b is displayed in place of calendar object 576 (asin FIG. 5LL). Selected calendar object 576 b is displayed with twohandles. In addition, a color of selected calendar object 576 b isdarkened (e.g., with respect to calendar object 576, FIG. 5LL), and ashadow is displayed such that calendar object 576 b appears to be liftedfrom the surface of user interface 575 (e.g., calendar object 576 bappears to be casting a shadow onto user interface 575). In someembodiments, in conjunction with visually indicating the selection ofcalendar object 576 (e.g., by displaying selected calendar object 576b), device 100 generates tactile output 578 (e.g., MicroTap High (270Hz) with a gain of 1.0).

FIG. 5NN illustrates a transition of user interface 575 from userinterface 575 in FIG. 5MM. In particular, FIG. 5NN illustrates a secondportion of user input 577 that includes lateral movement of user input577 (e.g., from the position of user input 577 as illustrated in FIG.5MM) toward a right edge of touch screen 112. In response to the lateralmovement of the second portion of user input 577, selected calendarobject 576 b is dragged toward the right edge of touch screen 112 inuser interface 575. In some embodiments, while selected calendar object576 b is dragged by movement of user input 577, calendar object 576 isdisplayed (e.g., a ghost image of the moving object that indicates anexisting or pre-movement position of the moving object, in someembodiments having the same appearance as unselected objects, such asthe unselected object “Have Lunch”). In some embodiments, while selectedcalendar object 576 b is dragged, selected calendar object 576 b snapsto one or more respective snap positions (e.g., corresponding torespective days or hours) when selected calendar object 576 b approachesrespective snap positions.

FIG. 5OO illustrates a transition of user interface 575 from userinterface 575 in FIG. 5NN, in response to a third portion of user input577 that includes further movement of user input 577 (e.g., movement ofcontinued contact corresponding to selected calendar object 576 b)toward the right edge of touch screen 112. In some embodiments, movementof user input 577 to a boundary (e.g., an edge) of the displayedcalendar interface, or to within a predetermined distance from aboundary of the displayed calendar interface, triggers updating thecalendar interface in accordance with the movement of user input 577.For example, in accordance with movement of user input 577 to a boundaryof the displayed calendar interface, or to within a predetermineddistance from the boundary of the displayed calendar interface, anadditional portion of the calendar interface that corresponds to (e.g.,is adjacent to) the boundary is displayed. In some embodiments, inconjunction with an additional portion of the calendar interface beingdisplayed, a portion of the calendar interface opposite the boundary isno longer displayed (e.g., a displayed date range or a displayed timerange of the calendar interface is shifted forward or backward by atleast one date or time interval). For example, in FIG. 5OO, inaccordance with the further movement of user input 577 toward the rightedge of touch screen 112, user input 577 moves within a predetermineddistance from a right edge of user interface 575 that corresponds to adate boundary for Saturday, September 6 (e.g., a date boundary betweenSaturday, September 6 and Sunday, September 7, not visible in userinterface 575 in FIG. 5OO).

In response to the further movement of the third portion of user input577 as illustrated in FIG. 5OO, FIG. 5PP illustrates a transition ofuser interface 575 from user interface 575 in FIG. 5OO. In particular,FIG. 5PP illustrates that, in response to user input 577, a date rangedisplayed in user interface 575 is shifted forward in time by one day(e.g., user interface 575 is updated to include display of Sunday,September 7 in the rightmost date column and to exclude display ofSunday, August 31 in the leftmost date column). In some embodiments, aposition of selected calendar object 576 b is maintained while the daterange is shifted forward (e.g., selected calendar object 576 b appearsto float above the surface of user interface 575 while the date rangeappears to slide forward underneath selected calendar object 576 b). Forexample, the position of selected calendar object 576 b is maintained inthe leftmost date column while user interface 575 is updated fromdisplaying Saturday, September 6 in the leftmost date column (FIG. 5OO)to displaying Sunday, September 7 in the leftmost date column, as shownin FIG. 5PP. Accordingly, selected calendar object 576 b snaps to arespective snap position on Sunday, September 7. In some embodiments, inaccordance with selected calendar object 576 b snapping to therespective snap position, device 100 generates tactile output 578 b(e.g., MicroTap High (270 Hz) with a gain of 0.4).

FIG. 5QQ illustrate a transition of user interface 575 from userinterface 575 in FIG. 5PP. In particular, FIG. 5QQ illustrates userinterface 575 in response to ceasing to detect user input 577 (e.g., inresponse to detecting a fourth portion of user input 577 that includesliftoff of the contact). In response to ceasing to detect user input577, device 100 visually indicates deselection of selected calendarobject 576 b (e.g., corresponding to dropping selected calendar object576 b in the respective position indicated in FIG. 5QQ) by ceasing todisplay calendar object 576 (e.g., the ghost object) in its previousposition (e.g., as shown in FIG. 5PP) and/or changing the appearance ofselected calendar object 576 b to appear unselected (e.g., by displayingcalendar object 576 in place of selected calendar object 576 b andceasing to display selected calendar object 576 b) such that calendarobject 576 appears to have been moved. In some embodiments, inaccordance with selected calendar object 576 b being dropped (e.g.,calendar object 576 being moved), device 100 generates tactile output578 c (e.g., MicroTap High (270 Hz) with a gain of 0.6).

FIGS. 5RR-5SS illustrate an example user interface for generatingtactile outputs with respect to rearranging array elements in an array,in accordance with some embodiments. In FIG. 5RR, user interface 580 aincludes an example home screen interface for displaying an array ofelements (e.g., application icons), including application icon 582,labeled “Maps,” and application icon 583, labeled “Stocks.” The array ofelements includes a plurality of predetermined snap positions thatcorrespond to a plurality of positions for displaying array elements. Asshown in FIG. 5RR, user input 581 (e.g., a contact) is detected at alocation on touch screen 112 that corresponds to application icon 582 inuser interface 580 a. In some embodiments, user interface 580 a isgenerated by using a UICollectionView class described in Appendix B.

FIG. 5SS illustrates a series of transitions of user interface 580 afrom user interface 580 a in FIG. 5RR. The series of transitions of userinterface 580 a includes user interfaces 580 b, 580 c, 580 d, and 580 ecorresponding to first, second, third, and fourth portions of user input581, respectively. In FIG. 5SS, time T=t₀ corresponds to user interface580 a as shown in and described above with reference to FIG. 5RR. Attime T=t₁, user interface 580 b illustrates that application icon 582 isselected in response to the first portion of user input 581 (e.g., along press gesture). In some embodiments, device 100 visually indicatesthat application icon 582 is selected (e.g., by shading application icon582 and/or enlarging application icon 582). Optionally, in someembodiments, in response to the first portion of user input 581selecting application icon 582, device 100 generates tactile output 584b (e.g., MicroTap High (270 Hz) with a gain of 1.0), as shown in tactileoutput graph 584, to indicate that application icon 582 has beenselected. In some embodiments, device 100 provides a visual indicationthat user interface 580 b corresponds to an array element rearrangementmode (e.g., by animating one or more application icons, such as by usinga shaking animation as indicated at 579) while application icon 582 isselected.

In some embodiments, while application icon 582 is selected, the secondportion of user input 581 is detected. For example, the second portionof user input 581 includes continued contact followed by movement of thecontact to the left toward “Stocks” application icon 583. At time T=t₂,user interface 580 c illustrates that, in accordance with the movementof user input 581 during the second portion of user input 581,application icon 582 moves to the left and passes over application icon583. Accordingly, application icon 582 is displayed over at least aportion of application icon 583.

At time T=t₃, user interface 580 d illustrates continued movement ofuser input 581 toward the left. In accordance with the continuedmovement of user input 581, application icon 582 moves into apredetermined snap position corresponding to the previous position ofunselected application icon 583 (e.g., as shown in user interface 580c). In some embodiments, the predetermined snap position correspondingto the previous position of unselected application icon 583 (e.g., asshown in user interface 580 a) becomes vacant. Accordingly, in someembodiments, unselected application icon 583 moves to the right into thepredetermined snap position vacated by application icon 583. In someembodiments, in conjunction with selected application icon 582 movinginto the predetermined snap position previously occupied by unselectedapplication icon 583, device 100 generates tactile output 584 d (e.g.,MicroTap High (270 Hz) with a gain of 1.0).

At time T=t₄, device 100 has ceased to detect user input 581 (e.g.,device 100 has detected the fourth portion of user input 581 includingliftoff of the contact) while selected application icon 582 is at thepredetermined snap position as shown in user interface 580 dcorresponding to time T=t₃. Accordingly, application icon 582 settlesinto and is displayed (e.g., as an unselected application icon in thearray) in the position previously occupied by application icon 583,while application icon 583 is displayed in the position previouslyoccupied by application icon 582 (e.g., application icon 582 andapplication icon 583 are displayed in switched positions). Optionally,in some embodiments, in response to application icon 582 setting intothe array, device 100 generates tactile output 584 e (e.g., MicroTapHigh (270 Hz) with a gain of 0.6) to indicate that application icon 582has been dropped.

FIGS. 5TT-5UU illustrate a tactile output that varies based on acharacteristic intensity of a user input. FIG. 5TT illustrates userinterface 590 a that includes an example user interface of a list ofe-mail summary items in an e-mail application (e.g., e-mail clientmodule 140, FIGS. 1A and 3) and a first portion of user input 591 (e.g.,a contact) detected at a location on touch screen 112 corresponding toe-mail 592 in the list of e-mail summary items. A characteristicintensity of the contact is above a detection threshold intensity IT0and below a hint threshold intensity ITH, as indicated by intensitymeter 596. In response to detecting user input 591, device 100 displaysa visual indication (e.g., highlighting) to indicate that e-mail 592 isselected.

FIG. 5UU illustrates a series of transitions of user interface 590 afrom user interface 590 a in FIG. 5TT. The series of transitions of userinterface 590 a include user interfaces 590 b, 590 c, and 590 dcorresponding to increases in a characteristic intensity of user input591. In FIG. 5UU, user interface 590 a as shown in and described abovewith reference to FIG. 5TT is displayed at time T=t₀. At time T=t₁, thecharacteristic intensity of user input 591 has increased to hintthreshold intensity IT_(H). Accordingly, device 100 displays a hintstate of e-mail 592, as shown in user interface 590 b. In someembodiments, the characteristic intensity of user input 591 continues toincrease between t₁ and t₂ toward light press threshold intensityIT_(L). In some embodiments, as the characteristic intensity of userinput 591 approaches light press threshold intensity IT_(L), device 100generates tactile output 594 a (e.g., as a tactile indication of thedetected increase in the characteristic intensity of user input 591). Insome embodiments, as the characteristic intensity of user input 591increases toward light press threshold intensity IT_(L), device 100gradually increases a characteristic of tactile output 594 a (e.g., anamplitude of tactile output 594 a increases from zero) as shown intactile output graph 594. In some embodiments, device 100 graduallyincreases the characteristic of tactile output 594 a as thecharacteristic intensity of user input 591 increases from zero. In someembodiments, device 100 gradually increases the characteristic oftactile output 594 a as the characteristic intensity of user input 591increases from hint threshold intensity IT_(H) (e.g., as shown intactile output graph 594).

At time T=t₂, the characteristic intensity of user input 591 crosseslight press threshold intensity IT_(L). In accordance with thecharacteristic intensity of user input 591 crossing light pressthreshold intensity IT_(L), user interface 590 c illustrates display ofa preview of information corresponding to e-mail 592 (e.g., a shortpreview of content of e-mail 592). In some embodiments, device 100generates tactile output 594 b (e.g., a discrete tap, such as a MicroTapCustom (200 Hz) with a gain of 1.0) when the characteristic intensity ofuser input 591 reaches light press threshold intensity IT_(L).

In some embodiments, the characteristic intensity of user input 591continues to increase between t₂ and t₃ toward deep press thresholdintensity IT_(D). In some embodiments, as the characteristic intensityof user input 591 approaches deep press threshold intensity, device 100generates tactile output 594 c (e.g., as a tactile indication of thedetected increase in the characteristic intensity of user input 591. Insome embodiments, device 100 gradually increases a characteristic oftactile output 594 c (e.g., an amplitude of tactile output 594 cincreases from zero) as shown in tactile output graph 594.

At time T=t₃, the characteristic intensity of user input 591 crossesdeep press threshold intensity IT_(D). In accordance with thecharacteristic intensity of user input 591 crossing deep press thresholdintensity IT_(D), user interface 590 d illustrates display of e-mail 592(e.g., the preview of content of e-mail 592 expands to a full screendisplay of e-mail 592). In some embodiments, device 100 generatestactile output 594 d (e.g., a discrete tap, such as a FullTap Custom(150 Hz) with a gain of 1.0) when the characteristic intensity of userinput 591 reaches deep press threshold IT_(D).

FIGS. 5VV-5YY illustrate example user interfaces and timing diagrams fordisplaying user interface changes and generating tactile outputscorresponding to user interface events.

FIG. 5VV illustrates an example user interface and timing diagram fordisplaying one or more user interface changes and generating a tactileoutput corresponding to a user interface event in response to a userinput (e.g., in response to direct manipulation of the user interface bya touch input). In some embodiments, display of the one or more userinterface changes is initiated immediately in response to the userinput. For example, in FIG. 5VV, user interface 570 c is a userinterface of a list of clocks as described above with reference to FIGS.5JJ-5KK. In accordance with upward movement of user input 571 detectedat time to, device 100 initiates display of changes to user interface570 c including movement of selected list element 572 upward into theposition previously occupied by unselected list element 573, andmovement of unselected list element 573 is downward into the positionpreviously occupied by selected list element 572. In addition, inaccordance with the upward movement of user input 571 detected at timeto, device 100 initiates generation of tactile output 574 dcorresponding to selected list element 572 moving into a predeterminedsnap position (e.g., corresponding to the previous position ofunselected list element 573), as shown in tactile output graph 574-2. Insome embodiments, the one or more changes to user interface 570 c aredisplayed at time T=t₁ as user interface 570 d, after a first timeinterval ΔT₁ (from t₀ to t₁) has passed. In some embodiments, generationof tactile output 574 d occurs at time T=t₂, after a second timeinterval ΔT₂ (from t₀ to t₂) associated with generation of tactileoutputs has elapsed. In some embodiments, the second time interval ΔT₂(e.g., 50-200 ms) is longer than the first time interval ΔT₁ (e.g., 5ms). In some embodiments, generation of tactile outputs is associatedwith a longer latency than a latency associated with displaying changesto the user interface.

In some embodiments, display of one or more user interface changesassociated with a user interface event (e.g., a user interface eventthat is independent of direct manipulation of the user interface inresponse to a touch input) is delayed such that the one or more userinterface changes is displayed concurrently with generation of a tactileoutput corresponding to the user interface event.

For example, FIG. 5WW illustrates display of user interface changes andgeneration of a tactile output in accordance with a user interface eventthat is not in response to a user input (e.g., a user interface eventthat is an alert or other notification that is independent of or not inresponse to direct manipulation of the user interface in response totouch input). At time T=t₀, user interface 510 a includes a userinterface for composing an e-mail message. Text 5101 represents apreviously (e.g., most recently) entered portion of text in userinterface 5100 a. At time T=t₁, movement of device 100 is detected. Insome embodiments, the movement of device 100 meets predefined inputpattern criteria (e.g., a pattern of shaking the device back and forth)corresponding to a user interface event that includes undoing apreviously performed operation (e.g., removing a portion of previouslyentered text). As illustrated in FIG. 5WW, the movement of device 100,detected at time t₁, corresponds to a user interface event to removepreviously entered text 5101 (e.g., by displaying undo prompt interface5100 b) and to generate tactile output 5102. In response to detectingthe movement of device 100 at time t₁, device 100 initiates, at time t₁,generation of tactile output 5102. In conjunction, in some embodiments,device 100 delays initiating display of the one or more changes to userinterface 510 a by delay time interval T_(delay) (e.g., device 100initiates displaying undo prompt interface 510 b at time t₂, after delaytime interval T_(delay) has elapsed). In some embodiments, generation oftactile output 5102 occurs at time t₃ after delay time interval ΔT₂associated with generation of tactile outputs has elapsed. Accordingly,in some embodiments, display of the one or more changes to userinterface 510 a (e.g., display of undo prompt interface 5100 b) issynchronized with generation of tactile output 5102 (e.g., at time t₃).

In another example, FIG. 5XX illustrates display of user interfacechanges and generation of a tactile output in accordance with a userinterface event that includes a notification that is independent ofdirect manipulation of the user interface in response to touch input. Attime T=t₀, user interface 5105 includes a user interface of a lockscreen of device 100. At time T=t₁, input 5106 (e.g., a notification ofa received calendar event invitation) is received (e.g., from aninformation source external to device 100). In response to receivinginput 5106 at time t₁, device 100 initiates, at time t₁, generation oftactile output 5107. In conjunction, device 100 delays initiatingdisplay of changes to user interface 5105 (e.g., display of calendarevent invitation affordance 5108 on user interface 5105) by delay timeinterval T_(delay) (e.g., device 100 initiates displaying calendar eventinvitation affordance 5108 at time t₂, after delay time intervalT_(delay) has elapsed). In some embodiments, generation of tactileoutput 5107 occurs at time t₃ after delay time interval ΔT₂ associatedwith generation of tactile outputs has elapsed. Accordingly, in someembodiments, display of the one or more changes to user interface 5105(e.g., display of calendar event invitation affordance 5108) issynchronized with generation of tactile output 5107 (e.g., at time t₃).

In yet another example, FIG. 5YY illustrates display of user interfacechanges and generation of a tactile output in accordance with a userinterface event that includes a notification initiated in accordancewith completion of a transaction initiated by a user. At time T=t₀, userinterface 511 a includes a user interface prompting a user to provideauthentication (e.g., a finger contact on a fingerprint sensor,illustrated as user input 5111) to complete a transaction. At time T=t₁,device 100 receives an input 5112 that includes authentication forcompletion of the transaction. In response to receiving input 5112 attime t₁, device 100 initiates, at time t₁, generation of tactile output5113. In conjunction, device 100 delays initiating display of changes touser interface 511 a (e.g., displaying user interface 511 b confirmingcompletion of the transaction) by delay time interval T_(delay) (e.g.,device 100 initiates displaying user interface 511 b at time t₂, afterdelay time interval T_(delay) has elapsed). In some embodiments,generation of tactile output 5113 occurs at time t₃ after delay timeinterval ΔT₂ associated with generation of tactile outputs has elapsed.Accordingly, in some embodiments, display of the one or more changes touser interface 511 a (e.g., display of user interface 5110 b) issynchronized with generation of tactile output 5113 (e.g., at time t₃).

FIG. 5ZZ illustrates timing diagrams for generating tactile outputsbased on states of tactile output generators, in accordance with someembodiments. In some embodiments, one or more tactile output triggersare received by tactile output controller 5120 (e.g., haptic feedbackcontroller 161, FIG. 1A). In some embodiments, in response to thetactile output triggers, tactile output controller 5120 transmitscontrol signals to one or more tactile output generators 5121 (e.g.,tactile output generator(s) 167, FIG. 1A). In response to the controlsignals, tactile output generator 5121 generates tactile outputs.

In some embodiments, in accordance with latency associated withgenerating tactile outputs, tactile output generator 5121 generatestactile outputs after a period of time after a tactile output trigger isreceived. The period of time between receipt of a tactile output triggerand generation of a corresponding tactile output varies in accordancewith a state of tactile output generator 5121. For example, asillustrated in FIG. 5ZZ, if a tactile output trigger is received whiletactile output generator 5121 is in an inactive state, a tactile outputis generated after a time period ΔT_(A) (e.g., 200 ms) after the tactileoutput trigger is received. In some embodiments, if a tactile outputtrigger is received while tactile output generator 5121 is in alow-latency state (e.g., a pre-warm state or an active state), a tactileoutput is generated after a time period that is less than ΔT_(A). Forexample, FIG. 5ZZ illustrates that if a tactile output trigger isreceived while tactile output generator 5121 is in a pre-warmed state, atactile output is generated after a time period ΔT_(B) (e.g., 100-150ms) after the tactile output trigger is received. In some embodiments,if a tactile output trigger is received while tactile output generator5121 is in an active state, a tactile output is generated after a timeperiod ΔT_(C) (e.g., 50 ms) after the tactile output trigger isreceived. In some embodiments, the time period associated with theactive state is less than the time period associated with the pre-warmstate.

FIGS. 5AAA-5CCC illustrate example user interfaces and timing forgenerating tactile outputs in accordance with user interface events fordifferent types of user interface events (e.g., success, failure, andwarning events).

FIG. 5AAA illustrates an example of one or more tactile outputsgenerated for success events. FIG. 5AAA includes user interface 5130 forentering authentication credentials for connecting to a wireless network(e.g., “HomeWireless”), displayed at time T=t₀. At time T=t₁, inaccordance with a determination that authentication of the passwordentered in user interface 5130 was successful, user interface 5131 isdisplayed (e.g., indicating successful connection to the requestedwireless network “HomeWireless”) and tactile output 5132 correspondingto a success event is generated (e.g., FullTap Medium (200 Hz) with again of 0.7 followed by a MiniTap High (270 Hz) with a gain of 1.0).

FIG. 5BBB illustrates an example of one or more tactile outputsgenerated for failure events. FIG. 5BBB includes user interface 5130 forentering authentication credentials for connecting to a wireless network(e.g., “HomeWireless”), displayed at time T=t₀. At time T=t₁, inaccordance with a determination that authentication of the passwordentered in user interface 5130 failed, user interface 5133 is displayed(e.g., indicating that an incorrect password was entered for therequested wireless network “HomeWireless”) and tactile output 5134corresponding to a failure event is generated (e.g., MiniTap High (270Hz) with a gain of 0.75-0.85, followed by a FullTap Medium (200 Hz) witha gain of 0.65, followed by a FullTap Low (150 Hz) with a gain of 0.75).

FIG. 5CCC illustrates an example of one or more tactile outputsgenerated for warning events. FIG. 5CCC includes user interface 5135 forcomposing an e-mail message, as described above with reference to FIG.5WW. At time T=t₀, movement of device 100 is detected (e.g., a patternof shaking the device back and forth corresponding to a request to undoa previous operation). At time T=t₁, in accordance with the request toundo the previous operation, undo prompt interface 510 b is displayed,as described above with reference to FIG. 5WW, and tactile output 5135corresponding to a warning event is generated (e.g., FullTap High (300Hz) with a gain of 0.9 followed by a FullTap Custom (270 Hz) with a gainof 0.9).

FIGS. 5DDD-5GGG illustrate an example user interface for generatingtactile outputs in accordance with magnitudes of operations performed inthe user interface in response to user input, in accordance with someembodiments. In particular, FIGS. 5DDD-5GGG illustrate tactile outputsgenerated during variable rate scrubbing, in accordance with movement ofa user input across boundaries.

FIG. 5DDD illustrates user interface 5140 for a media content playerthat includes slider control 5141, including adjustable progressindicator 5142 that indicates a current position in the content beingplayed on the device. In some embodiments, user interface 5140 includesa plurality of other media content player controls for controllingplayback of content in the media content player. FIG. 5DDD illustratesuser input 5143, initially detected at a location on touch screen 112that corresponds to an initial position of progress indicator 5142 inslider control 5141. FIG. 5DDD also illustrates movement of user input5143 away from its initial location (e.g., user input 5143 includes adrag gesture that includes an initial contact followed by movement ofthe contact) corresponding to slider control 5141. In some embodiments,boundaries 5144, 5145, and 5146 separate areas of user interface 5140that correspond to different scrubbing rates for adjusting the positionof progress indicator 5142 in slider control 5141. In some embodiments,boundaries 5144, 5145, and 5146 are not typically displayed to a user.In some embodiments, while user input 5143 is above boundary 5144, theposition of progress indicator 5142 in slider control 5141 moves by thesame amount as a horizontal component of movement of user input 5143 ontouch screen 112 that includes movement parallel to the slider control(sometimes referred to as “full-speed scrubbing”). In some embodiments,while user input 5143 is between boundary 5144 and boundary 5145, theposition of progress indicator 5142 in slider control 5141 moves by anamount that is a fraction (e.g., one-half or equivalently 50%) of thehorizontal component of movement of user input 5143 on touch screen 112that is parallel to slider control 5141 (sometimes referred to as“half-speed scrubbing”). In some embodiments, while user input 5143 isbetween boundary 5145 and boundary 5146, the position of progressindicator 5142 in slider control 5141 moves by an amount that is asmaller fraction (e.g., one-quarter or equivalently 25%) of thehorizontal component of movement of user input 5143 on touch screen 112that is parallel to slider control 5141 (sometimes referred to as“quarter-speed scrubbing”). In some embodiments, while user input 5143is below boundary 5146, the position of progress indicator 5142 inslider control 5141 moves by an amount that is an even smaller fraction(e.g., one-eighth or equivalently 12.5%) of the horizontal component ofmovement of user input 5143 on touch screen 112 that is parallel toslider control 5141 (sometimes referred to as “fine-speed scrubbing”).It is noted that the fractional scrubbing rates used here (50%, 25%, and12.5%) are merely examples, and that different scrubbing rates thatprogressively decrease as vertical distance between a user input andslider control 5141 increases may also be used.

In some embodiments, device 100 generates tactile outputs in accordancewith user input 5143 crossing respective boundaries. In someembodiments, the tactile outputs are generated in accordance with acharacteristic of user input 5143 that corresponds to an operation inuser interface 5140. For example, the tactile outputs are generated inaccordance with a velocity of user input 5143 when crossing a respectiveboundary that corresponds to a change in a scrubbing rate. In someembodiments, a stronger tactile output is generated when user input 5143crosses a respective boundary with a high velocity. In some embodiments,a lighter tactile output is generated when user input 5143 crosses arespective boundary with a low velocity.

FIG. 5EEE illustrates movement of user input 5143 downward and towardthe right with respect to its initial position (e.g., as shown in FIG.5DDD), away from slider control 5141 and crossing boundary 5144. Inaccordance with user input 5143 crossing boundary 5144, device 100generates tactile output 5148. In accordance with a velocity of themovement of user input 5143 being above a moderate velocity thresholdV_(M) and below a high velocity threshold V_(H), as indicated byvelocity of contact meter 5147, tactile output 5148 is generated as amoderate tactile output (e.g., MicroTap Medium (150 Hz) with a gain of0.4).

FIG. 5FFF illustrates further movement of user input 5143 downward andtoward the right with respect to its position as shown in FIG. 5EEE,further away from slider control 5141 and crossing boundary 5145. Inaccordance with user input 5143 crossing boundary 5145, device 100generates tactile output 5149. In accordance with the velocity of themovement of user input 5143 being above a high velocity threshold V_(F),as indicated by velocity of contact meter 5147, tactile output 5149 isgenerated as a strong tactile output (e.g., MicroTap Medium (150 Hz)with a gain of 0.7).

FIG. 5GGG illustrates even further movement of user input 5143 downwardand toward the right with respect to its position as shown in FIG. 5FFF,further away from slider control 5141 and crossing boundary 5146. Inaccordance with user input 5143 crossing boundary 5146, device 100generates tactile output 5150. In accordance with the velocity of themovement of user input 5143 being below the moderate velocity thresholdV_(M) and above a movement detection threshold V₀ (e.g., zero velocity),as indicated by velocity of contact meter 5147, tactile output 5150 isgenerated as a light tactile output (e.g., MicroTap Medium (150 Hz) witha gain of 0.1).

FIGS. 5HHH-5III illustrate timing diagrams for generating tactileoutputs in accordance with detecting a continuous user interaction andin accordance with a state of one or more tactile output generates usedto generate the tactile outputs. In particular, FIG. 5HHH illustratesslider 5160 having a position indicator 5161 in an initial position attime T=t₀. A tactile output generator for generating one or more tactileoutputs in accordance with position indicator 5161 of slider 5160crossing a respective marker on slider 5160 is initially in a low-powerstate, as shown in generator state graph 5164. In accordance with afirst portion of a user interaction that moves position indicator 5161to the position shown at time T=t₁, position indicator 5161 approachesbut does not cross marker 5162. The tactile output generator remains inthe low-power state (e.g., the state of the tactile output generator ismaintained or not changed). At time T=t₂, in accordance with a secondportion of the user interaction, position indicator 5161 crosses marker5162. Accordingly, generation of a tactile output is initiated at timet₂, as shown in tactile output graph 5165. In some embodiments, inaccordance with latency associated with the low-power state, tactileoutput 5163 is generated at time t=t₄, after time period ΔT_(A) (e.g.,200 ms) after the tactile output trigger is received at time t₂. Asshown in FIG. 5HHH, by time t₄ (at which tactile output 5163 isgenerated), position indicator 5161 has moved past marker 5162 inaccordance with continued movement of the user input.

In some embodiments, the state of the tactile output generator ischanged in anticipation of generating a tactile output, such as inaccordance with movement of a user input that approaches a threshold.For example, FIG. 5III illustrates slider 5160 having a positionindicator 5161 in an initial position at time T=t₀. A tactile outputgenerator for generating one or more tactile outputs in accordance withposition indicator 5161 of slider 5160 crossing a respective marker onslider 5160 is initially in a low-power state, as shown in generatorstate graph 5166. In accordance with a first portion of a userinteraction that moves position indicator 5161 to the position shown attime T=t₁, position indicator 5161 approaches but does not cross marker5162. In some embodiments, in response to position indicator 5161approaching marker 5162 (e.g., in anticipation of position indicator5161 crossing marker 5162, for which a tactile output will begenerated), the state of the tactile output generator is changed fromthe low-power state to the low-latency state (or alternatively, in someembodiments, the active state). At time T=t₂, in accordance with asecond portion of the user interaction, position indicator 5161 crossesmarker 5162. Accordingly, generation of a tactile output is initiated attime t₂, as shown in tactile output graph 5167. In some embodiments, inaccordance with latency associated with the low-latency state, tactileoutput 5168 is generated at time t=t₃, after time period ΔT_(B) (e.g.,100-150 ms) after the tactile output trigger is received at time t₂. Insome embodiments, time period ΔT_(B) associated with the low-latencystate is less than time period ΔT_(A) associated with the low-powerstate. As shown in FIG. 5III, the time t₃ at which tactile output 5168is generated occurs before time t₄ at which tactile output 5163 isgenerated as shown in FIG. 5HHH. In some embodiments, in accordance withchanging the state of the tactile output generator, tactile outputs aregenerated with lower latency. In addition, in FIG. 5III, positionindicator 5161 has moved past marker 5162 in accordance with continuedmovement of the user input to a lesser extent than in FIG. 5HHH.

FIG. SJJJ illustrates generation of a sequence of tactile outputs by anelectronic device (e.g., portable multifunction device 100, FIG. 1A)while enforcing a minimum time between repeated tactile outputs, inaccordance with some embodiments. Tactile output graph 517 a illustratestriggers for tactile outputs and generated tactile outputs associatedwith a first type of tactile output. Tactile output graph 517 billustrates triggers for tactile outputs and generated tactile outputsassociated with a second type of tactile output. At time t₁, generationof a tactile output of the first type is triggered (e.g., the devicegenerates the trigger, or the device receives a trigger from an externalsource), as shown in tactile output graph 5170 a. In response, thedevice generates tactile output 5171 at a point in time following thetrigger at t₁ and before a subsequent trigger at t₂. At time t₂, asecond instance of the first type of tactile output is triggered. Insome embodiments, the device determines that anticipated generation of atactile output in response to the trigger at t₂ is within apredetermined time from generation of tactile output 5171 (e.g., a mostrecent instance of the first type of tactile output that was generated).Accordingly, the device forgoes generation of the tactile output inresponse to the trigger at t₂.

Similarly, at time t₃, a third instance of the first type of tactileoutput is triggered, as shown in tactile output graph 5170 a. Inresponse, the device generates tactile output 5172 at a point in timefollowing the trigger at t₃ and before a subsequent trigger at t₄ (e.g.,in accordance with a determination that most recently generated tactileoutput 5171 of the first type of tactile output is at least thepredetermined time from anticipated generation of tactile output 5172).At time t₄, a fourth instance of the first type of tactile output istriggered, as shown in tactile output graph 5170 a. In addition, at timet₄, a first instance of a second type of tactile output is triggered, asshown in tactile output graph 5170 b. In some embodiments, the devicedetermines that anticipated generation of a tactile output of the firsttype in response to the trigger at t₄ is within the predetermined timefrom generation of tactile output 5172 (e.g., the most recent instanceof the first type of tactile output that was generated, determined withrespect to the trigger for the first type of tactile output at t₄).Accordingly, the device forgoes generation of the tactile output inresponse to the trigger for the first type of tactile output at t₄. Insome embodiments, the device generates tactile output 5176 in responseto the trigger for the second type of tactile output at t₄ (e.g.,without regard to timing of tactile outputs of the first type). In someembodiments, the device generates tactile output 5176 in accordance witha determination that generation of tactile output 5176 will be at leasta predetermined time from generation of a most recent instance of thesecond type of tactile output.

At time t₅, a fifth instance of the first type of tactile output istriggered, as shown in tactile output graph 5170 a. In response, thedevice generates tactile output 5173 at a point in time following thetrigger at t₅ and before a subsequent trigger at t₆ (e.g., in accordancewith a determination that most recently generated tactile output 5172 ofthe first type of tactile output is at least the predetermined time fromanticipated generation of tactile output 5173). At time t₆, a sixthinstance of the first type of tactile output is triggered, as shown intactile output graph 5170 a. In some embodiments, the device determinesthat anticipated generation of a tactile output in response to thetrigger at t₆ will be at least the predetermined time from generation oftactile output 5173 (e.g., the most recent instance of the first type oftactile output that was generated). Accordingly, the device generatestactile output 5174 in response to the trigger at t₆. Subsequently, attime t₇, a seventh instance of the first type of tactile output istriggered. In some embodiments, the device determines that anticipatedgeneration of a tactile output in response to the trigger at t₇ iswithin a predetermined time from generation of tactile output 5174(e.g., a most recent instance of the first type of tactile output thatwas generated). Accordingly, the device forgoes generation of thetactile output in response to the trigger at t₇.

At time t₈, an eighth instance of the first type of tactile output istriggered, as shown in tactile output graph 5170 a. In addition, at timet₈, a second instance of the second type of tactile output is triggered,as shown in tactile output graph 5170 b. In some embodiments, the devicedetermines that anticipated generation of a tactile output of the firsttype in response to the trigger at t₈ is at least the predetermined timefrom generation of tactile output 5174 (e.g., the most recent instanceof the first type of tactile output that was generated, determined withrespect to the trigger for the first type of tactile output at t₈).Accordingly, the device generates tactile output 5175 in response to thetrigger for the first type of tactile output at t₈. In addition, thedevice generates tactile output 5177 in response to the trigger for thesecond type of tactile output at t₈ (e.g., without regard to timing oftactile outputs of the first type). In some embodiments, the devicegenerates tactile output 5177 in accordance with a determination thatanticipated generation of tactile output 5177 is at least apredetermined time from generation of a most recent instance of thesecond type of tactile output (e.g., tactile output 5176).

FIG. 5KKK illustrates a conceptual flow diagram of method 5180 forcontrolling a state of one or more tactile output generators (e.g.,tactile output generator(s) 167, FIG. 1A). In some embodiments, method5180 is performed by an application-independent module of an electronicdevice (e.g., portable multifunction device 100, FIG. 1A).

In some embodiments, the application-independent module waits (5182) forinput from an application-specific module (e.g., a trigger forgenerating a tactile output, such as a user input that is associatedwith a tactile output). In some embodiments, while theapplication-independent module is waiting for input, the tactile outputgenerators are in an inactive state.

In some embodiments, the application-independent module receives (5184)an input from the application-specific module. In response to receivingthe input from the application-specific module, theapplication-independent module determines (5186) whether to prepare oneor more tactile output generators for generating tactile outputs (e.g.,whether the input meets tactile output generation criteria).

In accordance with a determination that one or more tactile outputgenerators should not be prepared (5186-No), the application-independentmodule returns to the wait state (5182) to wait for an additional input,and the tactile output generators are maintained in the inactive state.

In accordance with a determination that the one or more tactile outputgenerators should be prepared (5186-Yes) (e.g., in anticipation ofgenerating tactile outputs), the application-independent moduledetermines (5188) whether to prepare the tactile output generators bysetting the tactile output generators to a pre-warm state (e.g.,associated with a lower latency than the inactive state) or an activestate (e.g., associated with a lower latency than both the inactivestate and the pre-warm state).

In accordance with a determination that the tactile outputs should beset to the pre-warm state (5188-Pre-Warm), the application-independentmodule prepares the tactile output generators by setting (5190) thetactile output generators to the pre-warm state. In accordance with adetermination that the tactile outputs should be set to the active state(5188-Active), the application-independent module prepares the tactileoutput generators by setting (5192) the tactile output generators to theactive state.

Next, in some embodiments, the application-independent module determines(5194) whether to return the tactile output generators from the pre-warmor active state to the inactive state (e.g., whether to exit therespective preparation state). In accordance with a determination toexit the preparation state (5194-Yes) (e.g., the electronic device is ina battery conservation mode of operation, or a predetermined timeoutperiod has elapsed during which no tactile outputs were generated), theapplication-independent module returns to the wait state (5182). Inaccordance with a determination not to exit the preparation state(5194-No) (e.g., the predetermined timeout period has not elapsed sincepreparation of the tactile output generators was requested, or thepredetermined time period has not elapsed since generation of a mostrecent prior tactile output), the application-independent modulemaintains the respective preparation state of the tactile outputgenerators (e.g., the pre-warm state or the active state).

FIGS. 6A-6D are flow diagrams illustrating method 600 of generatingtactile outputs based on user interaction models in accordance with someembodiments. Method 600 is performed at an electronic device (e.g.,device 300, FIG. 3, or portable multifunction device 100, FIG. 1A) witha display, one or more input devices (e.g., a touch-sensitive surface,and/or one or more sensors to detect intensity of contacts with thetouch-sensitive surface), and one or more tactile output generators. Insome embodiments, the display is a touch-screen display and thetouch-sensitive surface is on or integrated with the display. In someembodiments, the display is separate from the touch-sensitive surface.Some operations in method 600 are, optionally, combined and/or the orderof some operations is, optionally, changed.

As described below, method 600 provides an enhanced way to providetactile outputs using an application-independent software module.Providing tactile outputs using application-independent software basedon information from application software provides common user interfaceframework that provides consistent user experience when various softwareapplications are used. Providing a common user interface frame work tothe user enhances the usability of such software applications and thedevice executing such software applications. In turn, this enhances theoperability of the device and makes the user-device interface moreefficient (e.g., by helping the user to provide proper inputs andreducing user mistakes and/or unintended operations whenoperating/interacting with the device). In addition, the method reducesthe size of software applications and makes execution of such softwareapplications faster. For battery-operated electronic devices, enabling auser to use software applications faster and more efficiently conservespower and increases the time between battery charges.

In some embodiments, the device stores (e.g., in memory) applicationsoftware (e.g., a first software application, such as application 1(136-1) in FIG. 1D) and application-independent software (e.g.,application-independent software module 260 in FIG. 1D) that isavailable for use by a plurality of software applications (e.g., thefirst software application, such as application 1 (136-1), and one ormore other software applications, such as application 2 (136-2) in FIG.1D) on the electronic device (e.g., the electronic device stores theplurality of software applications).

The device displays (602), on the display, a user interface for a firstsoftware application (e.g., the user interface for a settingsapplication shown in FIG. 5A). The user interface includes (604) aplurality of elements (e.g., switch object 502, slider object 504 andslider object 506) to which user interaction models from a plurality ofuser interaction models provided by an application-independent modulehave been assigned. The plurality of elements includes a first elementto which a first user interaction model has been assigned (e.g., sliderobject 502). The plurality of elements has content provided by anapplication-specific module for the first software application (e.g.,slider object 502 includes speaker icons provided by the settingsapplication).

In some embodiments, the device assigns (606) the first user interactionmodel to the first element using a predefined application programminginterface (API). For example, the device assigns a particular userinteraction model to the first element using an API for theapplication-independent software module.

The first user interaction model defines (608) how the user interfaceresponds to inputs directed to the first element of the user interface.For example, the first user interaction model defines whether or not togenerate a tactile output and/or a condition for generating the tactileoutput. In some embodiments, the first user interaction model alsodefines how the first element is visually updated in response to a userinput (e.g., moving a slider thumb, flipping on or off a switch, etc.).

While displaying the user interface, the device detects (610), via theone or more input devices, an input directed to the first element of theuser interface (e.g., input 509 directed to slider thumb 505 in FIG.5C).

In response to detecting the input (612), the device updates (614) theuser interface on the display based on characteristics of the input(e.g., magnitude of movement, direction of movement, start location,stop location, magnitude of intensity, rate of change of intensity). Forexample, a position of slider thumb 505 changes in response to a changeto input 509, as shown in FIG. 5D.

In some embodiments, the device updates (616) the user interface on thedisplay based on characteristics of the input and the first userinteraction model.

Also in response to detecting the input (612), and in accordance with adetermination that the input meets tactile output criteria specified bythe first user interaction model, the device generates (618) a firsttactile output corresponding to the input.

In some embodiments, the first element includes (620) a slider object(e.g., UISlider) generated at least in part based on informationprovided by the application-specific module for the first softwareapplication, and the first user interaction model specifies generatingtactile outputs (e.g., the first tactile output) in accordance with usermanipulation of the slider object with touch inputs.

In some embodiments, the device generates (622) a predefined modulationselection tactile output in response to a touch input that selects amodulation type for the slider object. For example, in FIG. 5B, apredefined tactile output is generated in response to input 503. Inanother example, a switch object or a list of options is configured tobe used to select a modulation type for a slider object (e.g., whetherthe slider object is a volume slider or a balance slider), and thedevice generates the predefined modulation selection tactile output inresponse to a touch input at a location that corresponds to the switchobject or the list of options.

In some embodiments, the first user interaction model specifies (624)generating the first tactile output in accordance with a modulation typeof the slider object and a parameter of the first software applicationmodulated in accordance with user manipulation of the slider object withtouch inputs. For example, as shown in FIGS. 5C-5E, in accordance with adetermination that the modulation type of the slider object is a volumeslider, a tactile output increases in its magnitude as a volumerepresented by the slider object increases. In accordance with adetermination that the modulation type of the slider object is a balanceslider, the balance of a tactile output is adjusted based on a locationof the slider thumb as shown in FIGS. 5G-5H.

In some embodiments, the first user interaction model specifies (626)generating a distinct tactile output in accordance with a movableportion of the slider object reaching an end position (e.g., in FIG. 5F,distinct tactile output 513 is generated in accordance with the sliderthumb reaching an end position of the slider).

In some embodiments, the first element includes (628) a switch object(e.g., UISwitch) generated at least in part based on informationprovided by the application-specific module for the first softwareapplication, and the first user interaction model specifies generatingthe first tactile output when the input corresponds to an action thatchanges a state of the switch object (e.g., the first input causes theswitch to be flipped from a prior state to a new state different fromthe prior state, such as from off to on or vice versa as shown in FIGS.5A-5B).

In some embodiments, the first element includes (630) a value selectionobject (e.g., using a UIPickerView class or a UIDatePicker classdescribed in Appendix B or their instances) generated at least in partbased on information provided by the application-specific module for thefirst software application, and the first user interaction modelspecifies generating the first tactile output when the input correspondsto an action that changes a value selected using the value selectionobject. For example, if the value selection object corresponds to ascrollable wheel or list having a sequence of different values, thedevice generates a sequence of tactile outputs as the first input causesthe value selection objection to scroll through the sequence ofdifferent values (e.g., the wheel of time shown in FIGS. 5K-5O). In oneexample, different values in the sequence are date values, for example,values for the year, month, day, hour or minute of a specified date ortime.

In some embodiments, the first element includes content that isrefreshed (632) in response to a pull down gesture reaching a refreshthreshold (e.g., using a UIRefreshControl class described in Appendix Bor its instance). The first user interaction model specifies generatinga predefined impact tactile output in accordance with a determinationthat the input corresponds to a pull down gesture reaching the refreshthreshold (e.g., content is refreshed in response to a pull down gesturereaching a refresh threshold as shown in FIGS. 5P-5T).

In some embodiments, the first element includes (634) a region or object(e.g., using a UIZoomEdgeFeedbackGenerator class or a UIScrollView classdescribed in Appendix B or their instances), the size of which changesin response to the input comprising a zoom gesture. The first userinteraction model specifies generating a predefined size limit tactileoutput in accordance with a determination that the size of the region orobject has reached a predefined limit (e.g., in accordance with adetermination that the size of the region or object, or equivalently azoom scaling factor, has reached a predefined limit). For example, thepredefined size limit tactile output is generated in accordance with adetermination that the size of the displayed web page has reached apredefined limit in response to a depinch gesture, as shown in FIGS.5Z-5CC.

In some embodiments, the first element includes (636) a table (e.g.,using a UITableView class described in Appendix B or its instance),having cells organized in rows and columns, generated at least in partbased on information provided by the application-specific module for thefirst software application (e.g., a calendar application provides datesand times of interest and/or calendar entries for the calendar viewillustrated in FIGS. 5LL-5QQ). The first user interaction modelspecifies generating the first tactile output when the input moves fromany cell in the table to any other cell in the table, or moves fromoutside the table to a cell inside the table (e.g., an input moves fromoutside the table to cell 576 b inside the table to select cell 576 b,as shown in FIGS. 5LL-5MM). In some embodiments, when the first inputcorresponds to a drag gesture for dragging a content item, the devicegenerates a snapping feedback tactile output when the first input movesan object into a cell in the table and the object “automatically” snapsinto place in the cell, where the table is displayed by an applicationand the application controls or specifies the a snapping action.

In some embodiments, the first element includes (638) a table (e.g.,using a UITableView class described in Appendix B or its instance),having cells organized in rows and columns, generated at least in partbased on information provided by the application-specific module for thefirst software application. The first user interaction model specifiesgenerating the first tactile output when two or more rows of the cellsare reordered in response to the input. For example, a tactile output isgenerated when two or more entries in a list are reordered as shown inFIG. 5KK. Similarly, in some embodiments, a tactile output is generatedwhen two or more entries in different rows are reordered in a table(e.g., by moving a calendar entry in a calendar view to another calendarentry in an adjacent row so that the two calendar entries arereordered).

In some embodiments, the input causes (640) the first softwareapplication to select an item, in a collection of distinct items (e.g.,using a UICollectionView class described in Appendix B or its instance),to be moved to a predefined position within the first element, and thefirst user interaction model specifies generating the first tactileoutput in conjunction with the selected item being moved to thepredefined position within the first element (e.g., a tactile output isgenerated when application icon 582 is moved to a predefined position,namely a position for an adjacent application icon 583 as shown in FIG.5SS). In some embodiments, the first user interaction model specifiesgenerating one or more tactile outputs in conjunction with the selecteditem moving to the predefined position in accordance with a predefinedsnapping behavior. For example, the selected item snaps into a placewhen a drag gesture brings the selected item within a predefineddistance of a predefined snap-in-place location (e.g., a grid locationin a predefined grid).

In some embodiments, the first user interaction model specifies (642)generating the first tactile output when the input or an objectmanipulated in accordance with the input approaches a next predefinedstate (e.g., a force-press gesture), and generating a second tactileoutput, distinct from the first tactile output, when the input or anobject manipulated in accordance with the input reaches the nextpredefined state (e.g., in FIG. 5UU, the device generates tactile output594 a when the input intensity approaches the intensity thresholdIT_(L), and generates tactile output 594 b when the input intensityreaches the intensity threshold IT_(L)). In some embodiments, the firstbehavior model specifies generating a sequence of distinct tactileoutputs as the intensity of the first touch input reaches each of asequence of intensity thresholds.

In accordance with a determination that the input does not meet thetactile output criteria specified by the first user interaction model,the device forgoes (644) generation of the first tactile output (e.g.,as shown in FIG. 5AA, when the size of the region or object, orequivalently a zoom scaling factor, has not reached a predefined limit,the first tactile output, such as tactile output 564 shown in FIG. 5CC,is not generated).

In some embodiments, the plurality of elements includes (646) a secondelement to which a second user interaction model from the plurality ofuser interaction models provided by the application-independent modulehas been assigned. While displaying the user interface, the devicedetects, via the one or more input devices, a second input directed tothe second element of the user interface; and, in response to detectingthe second input directed to the second element of the user interface:updates the user interface on the display based on characteristics ofthe detected second input; in accordance with a determination that thesecond input meets tactile output criteria specified by the second userinteraction model, generates a second tactile output corresponding tothe second input (e.g., a tap gesture or a horizontal swipe gesture onswitch object 502 flips a state of switch object 502 and initiatesgeneration of the second tactile output indicating that the state ofswitch object 502 has been flipped); and, in accordance with adetermination that the second input does not meet the tactile outputcriteria specified by the second user interaction model, forgoesgeneration of the second tactile output (e.g., a vertical swipe gestureon switch object 502 does not flip the state of switch object 502 andthe second tactile output is not generated).

It should be understood that the particular order in which theoperations in FIGS. 6A-6D have been described is merely an example andis not intended to indicate that the described order is the only orderin which the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein. Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,methods 800, 1000, 1200, 1400, 1600, 1800, and 2000) are also applicablein an analogous manner to method 600 described above with respect toFIGS. 6A-6D. For example, the tactile outputs, slider objects, switchobjects, value selection objects, content refreshing operations, zoomingoperations, scrolling operations, table manipulations, and selecting andmoving user interface objects described above with reference to method600 optionally have one or more of the characteristics of the tactileoutputs, slider objects, switch objects, value selection objects,content refreshing operations, zooming operations, scrolling operations,table manipulations, and selecting and moving user interface objectsdescribed herein with reference to other methods described herein (e.g.,methods 800, 1000, 1200, 1400, 1600, 1800, and 2000). For brevity, thesedetails are not repeated here.

The foregoing description, for purpose of explanation, has beendescribed with reference to specific embodiments. However, theillustrative discussions above are not intended to be exhaustive or tolimit the invention to the precise forms disclosed. Many modificationsand variations are possible in view of the above teachings. For example,in accordance with some embodiments, the device displays, on thedisplay, a user interface for a first software application. The userinterface is generated by populating a first user interaction templatefrom a plurality of user interaction templates provided by anapplication-independent module with content provided by anapplication-specific module for the first software application. Thefirst user interaction template defines how the user interface respondsto inputs directed to the user interface. While displaying the userinterface, the device detects, via the one or more input devices, aninput directed to the user interface. In response to detecting theinput, the device updates the user interface on the display based oncharacteristics of the detected input (e.g., magnitude of movement,direction of movement, start location, stop location, magnitude ofintensity, rate of change of intensity) and the first user interactiontemplate; in accordance with a determination that the input meetstactile output criteria specified by the first user interactiontemplate, generates a first tactile output corresponding to the input;and, in accordance with a determination that the input does not meet thetactile output criteria specified by the first user interactiontemplate, forgoes generation of the first tactile output.

The operations in the information processing methods described aboveare, optionally implemented by running one or more functional modules ininformation processing apparatus such as general purpose processors(e.g., as described above with respect to FIGS. 1A and 3) or applicationspecific chips.

In accordance with some embodiments, FIG. 7 shows a functional blockdiagram of electronic device 700 configured in accordance with theprinciples of the various described embodiments. The functional blocksof the device are, optionally, implemented by hardware, software, or acombination of hardware and software to carry out the principles of thevarious described embodiments. It is understood by persons of skill inthe art that the functional blocks described in FIG. 7 are, optionally,combined or separated into sub-blocks to implement the principles of thevarious described embodiments. Therefore, the description hereinoptionally supports any possible combination or separation or furtherdefinition of the functional blocks described herein.

As shown in FIG. 7, electronic device 700 includes display unit 702(e.g., including display 112) configured to display user interfaces, oneor more input device unit(s) 704 configured to receive inputs (e.g.,touch inputs on a surface, such as a display surface of display unit702), and processing unit 708 coupled with display unit 702 and the oneor more input device unit(s) 704. In some embodiments, electronic device700 also includes one or more tactile output generator unit(s) 706configured to generate one or more tactile outputs, also coupled toprocessing unit 708. In some embodiments, processing unit 708 includesone or more of the following sub-units: display enabling unit 710,detecting unit 712, tactile output enabling unit 714, assigning unit716, and determining unit 718.

In some embodiments, processing unit 708 is configured to enable display(e.g., using display enabling unit 702), on display unit 702, of a userinterface for a first software application, wherein the user interfaceincludes a plurality of elements to which user interaction models from aplurality of user interaction models provided by anapplication-independent module have been assigned (e.g., using assigningunit 716), the plurality of elements include a first element to which afirst user interaction model has been assigned, and the plurality ofelements have content provided by an application-specific module for thefirst software application. In some embodiments, the first userinteraction model defines how the user interface responds to inputsdirected to the first element of the user interface. While enablingdisplay of the user interface, processing unit 708 is configured todetect (e.g., using detecting unit 712), via input device unit(s) 704,an input directed to the first element of the user interface. Inresponse to detecting the input, processing unit 708 updates (e.g.,using display enabling unit 710) the user interface on the display unitbased on characteristics of the detected input. In accordance with adetermination (e.g., made using determining unit 718) that the inputmeets tactile output criteria specified by the first user interactionmodel, processing unit 708 enables generation of a first tactile outputcorresponding to the input (e.g., using tactile output enabling unit714). In accordance with a determination (e.g., made using determiningunit 718) that the input does not meet the tactile output criteriaspecified by the first user interaction model, processing unit 708forgoes enabling generation of the first tactile output (e.g., usingtactile output enabling unit 714).

In some embodiments, processing unit 708 updates (e.g., using displayenabling unit 710) the user interface on display unit 702 based oncharacteristics of the detected input and the first user interactionmodel. In some embodiments, processing unit 708 assigns (e.g., usingassigning unit 716) the first user interaction model to the firstelement using a predefined application programming interface (API).

In some embodiments, the plurality of elements includes a second elementto which a second user interaction model from the plurality of userinteraction models provided by the application-independent module hasbeen assigned (e.g., using assigning unit 716), and while displaying theuser interface, processing unit 708 detects (e.g., using detecting unit712), via the one or more input device units 704, a second inputdirected to the second element of the user interface. In response todetecting the second input directed to the second element of the userinterface, processing unit 708 updates (e.g., using display enablingunit 710) the user interface on the display unit based oncharacteristics of the detected second input. In accordance with adetermination (e.g., made using determining unit 718) that the secondinput meets tactile output criteria specified by the second userinteraction model, processing unit 708 enables (e.g., using tactileoutput enabling unit 714) generation of a second tactile outputcorresponding to the second input, and in accordance with adetermination (e.g., made using determining unit 718) that the secondinput does not meet the tactile output criteria specified by the seconduser interaction model, processing unit 708 forgoes enabling generationof the second tactile output.

In some embodiments, the first element includes a slider objectgenerated at least in part based on information provided by theapplication-specific module for the first software application, and thefirst user interaction model specifies generating tactile outputs (e.g.,using tactile output enabling unit 714) in accordance with usermanipulation of the slider object with touch inputs. In someembodiments, processing unit 708 is further configured to enablegeneration of a predefined modulation selection tactile output (e.g.,using tactile output enabling unit 714) in response to a touch inputthat selects a modulation type for the slider object.

In some embodiments, the first user interaction model specifiesgenerating the first tactile output (e.g., using tactile output enablingunit 714) in accordance with a modulation type of the slider object anda parameter of the first software application modulated in accordancewith user manipulation of the slider object with touch inputs.

In some embodiments, the first user interaction model specifiesgenerating a distinct tactile output (e.g., using tactile outputenabling unit 714) in accordance with a movable portion of the sliderobject reaching an end position.

In some embodiments, the first element includes a switch objectgenerated at least in part based on information provided by theapplication-specific module for the first software application, and thefirst user interaction model specifies generating the first tactileoutput (e.g., using tactile output enabling unit 714) when the firstinput corresponds to an action that changes a state of the switchobject.

In some embodiments, the first element includes a value selection objectgenerated at least in part based on information provided by theapplication-specific module for the first software application, and thefirst user interaction model specifies generating the first tactileoutput (e.g., using tactile output enabling unit 714) when the firstinput corresponds to an action that changes a value selected using thevalue selection object.

In some embodiments, the first element includes content that isrefreshed in response to a pull down gesture reaching a refreshthreshold, and the first user interaction model specifies generating apredefined impact tactile output (e.g., using tactile output enablingunit 714) in accordance with a determination (e.g., made usingdetermining unit 718) that the first input corresponds to a pull downgesture reaching the refresh threshold.

In some embodiments, the first element includes a region or object, thesize of which changes in response to the first input comprising a zoomgesture, and the first user interaction model specifies generating apredefined size limit tactile output (e.g., using tactile outputenabling unit 714) in accordance with a determination (e.g., made usingdetermining unit 718) that the size of the region or object has reacheda predefined limit.

In some embodiments, the first element includes a table, having cellsorganized in rows and columns, generated at least in part based oninformation provided by the application-specific module for the firstsoftware application, and the first user interaction model specifiesgenerating the first tactile output (e.g., using tactile output enablingunit 714) when the first input moves from any cell in the table to anyother cell in the table, or moves from outside the table to a cellinside the table.

In some embodiments, the first element includes a table, having cellsorganized in rows and columns, generated at least in part based oninformation provided by the application-specific module for the firstsoftware application, and the first user interaction model specifiesgenerating the first tactile output (e.g., using tactile output enablingunit 714) when two or more rows of the cells are reordered in responseto the first input.

In some embodiments, the first input causes the first softwareapplication to select an item, in a collection of distinct items, to bemoved to a predefined position within the first element, and the firstuser interaction model specifies generating the first tactile output(e.g., using tactile output enabling unit 714) in conjunction with theselected item being moved to the predefined position within the firstelement.

In some embodiments, the first user interaction model specifiesgenerating (e.g., using tactile output enabling unit 714) the firsttactile output when the first input or an object manipulated inaccordance with the first input approaches a next predefined state, andgenerating a second tactile output, distinct from the first tactileoutput, when the first input or an object manipulated in accordance withthe first input reaches the next predefined state.

The operations described above with reference to FIGS. 6A-6D are,optionally, implemented by components depicted in FIGS. 1A-1B or FIG. 7.For example, detection operation 610, updating operation 614, andgenerating operation 618 are, optionally, implemented by event sorter170, event recognizer 180, and event handler 190. Event monitor 171 inevent sorter 170 detects a contact on touch-sensitive display 112, andevent dispatcher module 174 delivers the event information toapplication 136-1. A respective event recognizer 180 of application136-1 compares the event information to respective event definitions186, and determines whether a first contact at a first location on thetouch-sensitive surface (or whether rotation of the device) correspondsto a predefined event or sub-event, such as selection of an object on auser interface, or rotation of the device from one orientation toanother. When a respective predefined event or sub-event is detected,event recognizer 180 activates an event handler 190 associated with thedetection of the event or sub-event. Event handler 190 optionally usesor calls data updater 176 or object updater 177 to update theapplication internal state 192. In some embodiments, event handler 190accesses a respective GUI updater 178 to update what is displayed by theapplication. Similarly, it would be clear to a person having ordinaryskill in the art how other processes can be implemented based on thecomponents depicted in FIGS. 1A-1B.

FIGS. 8A-8D are flow diagrams illustrating method 800 of synchronizingtactile outputs and user interface changes in accordance with someembodiments. Method 800 is performed at an electronic device (e.g.,device 300, FIG. 3, or portable multifunction device 100, FIG. 1A) witha display, a touch-sensitive surface, and one or more tactile outputgenerators. In some embodiments, the display is a touch-screen displayand the touch-sensitive surface is on or integrated with the display. Insome embodiments, the display is separate from the touch-sensitivesurface. Some operations in method 800 are, optionally, combined and/orthe order of some operations is, optionally, changed.

As described below, method 800 provides an intuitive way to providetactile outputs in conjunction with user interface events. Providingtactile outputs takes longer than updating a user interface. Thus, userinterface events (e.g., time-flexible user interface events) are delayedso that the delayed user interface events are synchronized withgeneration of tactile outputs. Synchronizing generation of tactileoutputs and user interface updates provides intuitive user interfaces.On the other hand, for time-critical user interface events, userinterface events are not delayed, which maintains the responsiveness ofthe device for such time-critical user interface events. The methodreduces the number, extent, and/or nature of the inputs from a user wheninteracting with user interfaces, thereby creating a more efficienthuman-machine interface. For battery-operated electronic devices,enabling a user to interact with user interfaces faster and moreefficiently conserves power and increases the time between batterycharges.

The device displays (802), on the display, a user interface (e.g., userinterface 5105 as illustrated for T=t₀ in FIG. 5XX).

The device detects (804) an occurrence of a first condition (e.g., auser input, a scheduled event such as an alarm or a timer, or anexternally triggered event such as an incoming notification) thattriggers a first user interface event that is associated with a tactileoutput (e.g., receiving a calendar event invitation as illustrated forT=t₁ in FIG. 5XX). Generating the first user interface event includes(806) displaying one or more changes to the user interface (e.g.,display of calendar event invitation affordance 5108 on user interface5105 in FIG. 5XX).

In some embodiments, the occurrence of the first condition includes(808) detected movement of the electronic device that meets predefinedinput pattern criteria (e.g., a pattern of shaking the device back andforth as illustrated in FIG. 5WW for T=t₁) and the one or more changesto the user interface include undoing a previously performed operation(e.g., undoing an auto-correction, redisplaying one or more deletedcharacters, removing one or more previously typed characters, undoingdeletion of an item, and/or displaying an undo operation affordance asillustrated in FIG. 5WW for T=t₃).

The device will not be able to generate (810) the tactile outputassociated with the first user interface event for a respective amountof time (e.g., due to other tactile outputs being generated and/or thetactile output generators not being ready to generate the tactile outputbecause the tactile output generators are in a low-power state). In someembodiments, the device is not configured to generate the tactile outputassociated with the first user interface event for the respective amountof time (e.g., it takes at least the respective amount of time, such asT_(delay) illustrated in FIG. 5WW or more, from a point of time when thetactile output associated with the first user interface event isrequested or initiated to when the tactile output is generated).

In response to detecting the occurrence of the first condition, inaccordance with a determination that the first user interface eventcorresponds to a first user interface event category (812) (e.g., thefirst user interface event is not in response to a user input such as analert or other notification that is not direct manipulation of the userinterface in response to touch input), the device delays (814)generating the first user interface event for at least the respectiveamount of time (e.g., in FIG. 5WW, the device delays generating thefirst user interface event for T_(delay)).

In some embodiments, the device, when the respective amount of time is afirst amount of time, delays (816) displaying the one or more changes tothe user interface for the first amount of time, and when the respectiveamount of time is a second amount of time, delays displaying the one ormore changes to the user interface for the second amount of time (e.g.,the device delays displaying the one or more changes to the userinterface based on the time it takes to generate a tactile output).

In some embodiments, the determination that the first user interfaceevent corresponds to the first user interface event category includes(818) a determination that the first user interface event is anotification event that does not correspond to a user input currentlydetected on the touch-sensitive surface (e.g., receiving a calendarevent invitation as illustrated for T=t₁ in FIG. 5XX). In someembodiments, the determination that the first user interface eventcorresponds to the first user interface event category includes adetermination that the first user interface event is independent of anyuser input on the touch-sensitive surface that is detected at the sametime as, or immediately preceding, the notification event.

In some embodiments, the determination that the first user interfaceevent corresponds to the first user interface event category includes(820) a determination that the first user interface event is anotification event initiated in accordance with receiving electronicallydelivered information from an information source external to theelectronic device (e.g., receiving a calendar event invitation asillustrated for T=t₁ in FIG. 5XX).

In some embodiments, the determination that the first user interfaceevent corresponds to the first user interface event category includes(822) a determination that the first user interface event is anotification event initiated in accordance with receiving electronicallydelivered information from an application executed by the electronicdevice (e.g., receiving a calendar event invitation as illustrated forT=t₁ in FIG. 5XX) and independent of any user input on thetouch-sensitive surface that is detected at the same time as, orimmediately preceding, the notification event.

In some embodiments, the determination that the first user interfaceevent corresponds to the first user interface event category includes(824) a determination that the first user interface event is anotification event initiated in accordance with completion of atransaction initiated by a user of the electronic device (e.g.,transmission of payment credentials via a short range wirelesscommunication signal to a payment terminal such as an NFC payment,successful authorization of a payment via a biometric sensor such as afingerprint sensor, or receiving authorization of a payment by a remotepayment server as shown in FIG. 5YY).

In some embodiments, the determination that the first user interfaceevent corresponds to the first user interface event category includes(826) a determination that the first user interface event is provided toan application programming interface (API) with a value indicating thatthe one or more changes to the user interface can be delayed until thedevice is ready to generate the tactile output (e.g., the first userinterface event is provided from an application to the applicationprogramming interface (and to application-independent software module260 illustrated in FIG. 1D) with information indicating whether thefirst user interface event is a flexible event or an inflexible event).

After delaying generating the first user interface event for at leastthe respective amount of time, the device displays (828) the one or morechanges to the user interface and generating the tactile output that isassociated with the first user interface event. The display of the oneor more changes to the user interface is synchronized (830) with thegeneration of the tactile output that is associated with the first userinterface event (e.g., so that the one or more changes to the userinterface are displayed with the generation of the first tactile outputas shown in FIG. 5WW) (e.g., using a prepareWithCompletionHandlerfunction described in Appendix B).

In some embodiments, in response to detecting the occurrence of thefirst condition, in accordance with a determination that the first userinterface event corresponds to a second user interface event category(e.g., the first user interface event is in response to a user input,such as changes to the user interface that correspond to directmanipulation of the user interface in response to touch input), thedevice displays (832) the one or more changes to the user interfacebefore the respective amount of time has elapsed and generates thetactile output that is associated with the one or more changes in theuser interface after the respective amount of time has elapsed (e.g., inFIG. 5VV, the one or more changes to the user interface are displayedwithout the delay to provide a prompt visual feedback).

In some embodiments, the first user interface event category includes(834) events where the respective amount of time for the one or morechanges to the user interface is flexible (e.g., user interface eventsthat are independent of immediately preceding user inputs). The seconduser interface event category includes events where the respectiveamount of time for the one or more changes to the user interface isinflexible (e.g., user interface events initiated by user inputs).

In some embodiments, the first user interface event category includes(836) events where the one or more changes to the user interface do notsimulate direct manipulation of the user interface by user inputs (e.g.,receiving a calendar event invitation as illustrated for T=t₁ in FIG.5XX). The second user interface event category includes events where theone or more changes to the user interface simulate direct manipulationof the user interface by user inputs (e.g., user inputs that triggeredthe first user interface event, such movement of a contact thatcorresponds to movement of a user interface object in the user interfaceas shown in FIG. 5VV or an increase in intensity of a contact thatcorresponds to changing the appearance of a user interface object basedon the pressure of the contact).

In some embodiments, the device detects (838) a user input on thetouch-sensitive surface that triggers a second user interface event thatis associated with a tactile output (e.g., user input 571 in FIG. 5VV).Generating the second user interface event includes displaying one ormore changes to the user interface. In response to detecting the userinput on the touch-sensitive surface, the device displays the one ormore changes to the user interface for the second user interface eventand generates the tactile output that is associated with the second userinterface event (e.g., list element 572 and list element 573 arereordered in FIG. 5VV and a corresponding tactile output is generated).A first time interval between the display of the one or more changes tothe user interface for the first user interface event and the generationof the tactile output that is associated with the first user interfaceevent is less than a second time interval between the display of the oneor more changes to the user interface for the second user interfaceevent and the generation of the tactile output that is associated withthe second user interface event (e.g., in some embodiments, the firsttime interval between the display of one or more changes to the userinterface for the first user interface event and the generation of thetactile output that is associated with the first user interface event inFIG. 5WW is close to zero, and is less than the second time intervalbetween t₁ and t₂ in FIG. 5VV).

It should be understood that the particular order in which theoperations in FIGS. 8A-8D have been described is merely an example andis not intended to indicate that the described order is the only orderin which the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein. Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,methods 600, 1000, 1200, 1400, 1600, 1800, and 2000) are also applicablein an analogous manner to method 800 described above with respect toFIGS. 8A-8D. For example, the tactile outputs, user interface elements,user interface events, user interface event categories, notificationevents, and synchronization of tactile outputs with user interfaceevents described above with reference to method 800 optionally have oneor more of the characteristics of the tactile outputs, user interfaceelements, user interface events, user interface event categories,notification events, and synchronization of tactile outputs with userinterface events described herein with reference to other methodsdescribed herein (e.g., methods 600, 1000, 1200, 1400, 1600, 1800, and2000). For brevity, these details are not repeated here.

The operations in the information processing methods described aboveare, optionally implemented by running one or more functional modules ininformation processing apparatus such as general purpose processors(e.g., as described above with respect to FIGS. 1A and 3) or applicationspecific chips.

In accordance with some embodiments, FIG. 9 shows a functional blockdiagram of electronic device 900 configured in accordance with theprinciples of the various described embodiments. The functional blocksof the device are, optionally, implemented by hardware, software, or acombination of hardware and software to carry out the principles of thevarious described embodiments. It is understood by persons of skill inthe art that the functional blocks described in FIG. 9 are, optionally,combined or separated into sub-blocks to implement the principles of thevarious described embodiments. Therefore, the description hereinoptionally supports any possible combination or separation or furtherdefinition of the functional blocks described herein.

As shown in FIG. 9, electronic device 900 includes display unit 902(e.g., including display 112) configured to display user interfaces,touch-sensitive surface unit 904 configured to receive touch inputs(e.g., on a display surface of display unit 902), one or more tactileoutput generator unit(s) 906 configured to generate one or more tactileoutputs, and processing unit 908 coupled with display unit 902,touch-sensitive surface unit 904, and one or more tactile outputgenerator unit(s) 906. In some embodiments, processing unit 908 includesone or more of the following sub-units: display enabling unit 910,detecting unit 912, delay unit 914, tactile output enabling unit 916,and determining unit 918.

In some embodiments, processing unit 908 is configured to enable display(e.g., using display enabling unit 910), on display unit 902, of a userinterface. Processing unit 908 detects (e.g., using detecting unit 912)an occurrence of a first condition that triggers a first user interfaceevent that is associated with a tactile output, wherein generating thefirst user interface event includes enabling display (e.g., usingdisplay enabling unit 910) of one or more changes to the user interface.In some embodiments, the device (e.g., device 900) will not be able togenerate the tactile output (e.g., using tactile output generator units906) associated with the first user interface event for a respectiveamount of time, and in response to detecting (e.g., using detecting unit912) the occurrence of the first condition, in accordance with adetermination (e.g., made using determining unit 918) that the firstuser interface event corresponds to a first user interface eventcategory, processing unit 908 enables generation (e.g., using tactileoutput enabling unit 916) and delays (e.g., using delay unit 914)generation of the first user interface event for at least the respectiveamount of time. After delaying (e.g., using delay unit 914) generationof the first user interface event for at least the respective amount oftime, processing unit 908 enables display (e.g., using display enablingunit 910) of the one or more changes to the user interface, wherein thedisplay of the one or more changes to the user interface is synchronizedwith the generation of the tactile output that is associated with thefirst user interface event.

In some embodiments, processing unit 908 is further configured to, whenthe respective amount of time is a first amount of time, delay (e.g.,using delay unit 914) displaying the one or more changes to the userinterface for the first amount of time, and when the respective amountof time is a second amount of time, delay (e.g., using delay unit 914)displaying the one or more changes to the user interface for the secondamount of time.

In some embodiments, processing unit 908 is configured to, in responseto detecting the occurrence of the first condition, in accordance with adetermination (e.g., made using determining unit 918) that the firstuser interface event corresponds to a second user interface eventcategory, enable display of the one or more changes to the userinterface (e.g., using display enabling unit 910) before the respectiveamount of time has elapsed, wherein the tactile output (e.g., usingtactile feedback unit 918) that is associated with the one or morechanges in the user interface is generated after the respective amountof time has elapsed.

In some embodiments, the first user interface event category includesevents where the timing of the one or more changes to the user interfaceis flexible, and the second user interface event category includesevents where the timing of the one or more changes to the user interfaceis inflexible.

In some embodiments, the first user interface event category includesevents where the one or more changes to the user interface do notsimulate direct manipulation of the user interface by user inputs, andthe second user interface event category includes events where the oneor more changes to the user interface simulate direct manipulation ofthe user interface by user inputs.

In some embodiments, processing unit 908 is configured to detects a userinput on touch-sensitive surface unit 904 that triggers a second userinterface event that is associated with a tactile output, whereingenerating the second user interface event includes enabling display(e.g., using display enabling unit 910) of one or more changes to theuser interface. In response to detecting the user input ontouch-sensitive surface unit 904, processing unit 908 is configured toenable display (e.g., using display enabling unit 910) of the one ormore changes to the user interface for the second user interface eventand enable generation of the tactile output (e.g., using tactile outputenabling unit 916) that is associated with the second user interfaceevent, wherein a first time interval between the display of the one ormore changes to the user interface for the first user interface eventand the generation of the tactile output that is associated with thefirst user interface event is less than a second time interval betweenthe display of the one or more changes to the user interface for thesecond user interface event and the generation of the tactile outputthat is associated with the second user interface event.

In some embodiments, the occurrence of the first condition includesdetected movement of electronic device 900 that meets predefined inputpattern criteria and the one or more changes to the user interfaceinclude undoing a previously performed operation.

In some embodiments, the determination that the first user interfaceevent corresponds to the first user interface event category includes adetermination (e.g., made using determining unit 918) that the firstuser interface event is a notification event that does not correspond toa user input currently detected on touch-sensitive surface unit 904.

In some embodiments, the determination that the first user interfaceevent corresponds to the first user interface event category includes adetermination (e.g., made using determining unit 918) that the firstuser interface event is a notification event initiated in accordancewith receiving electronically delivered information from an informationsource external to electronic device 900.

In some embodiments, the determination that the first user interfaceevent corresponds to the first user interface event category includes adetermination (e.g., made using determining unit 918) that the firstuser interface event is a notification event initiated in accordancewith receiving electronically delivered information from an applicationexecuted by electronic device 900 and independent of any user input ontouch-sensitive surface unit 904 that is detected at the same time as,or immediately preceding, the notification event.

In some embodiments, the determination that the first user interfaceevent corresponds to the first user interface event category includes adetermination (e.g., made using determining unit 918) that the firstuser interface event is a notification event initiated in accordancewith completion of a transaction initiated by a user of electronicdevice 900.

In some embodiments, the determination that the first user interfaceevent corresponds to the first user interface event category includes adetermination (e.g., made using determining unit 918) that the firstuser interface event is provided to an application programming interface(API) with a value indicating that the one or more changes to the userinterface can be delayed until device 900 is ready to generate thetactile output.

The operations described above with reference to FIGS. 8A-8D are,optionally, implemented by components depicted in FIGS. 1A-1B or FIG. 9.For example, detection operation 804, delaying operation 814, anddisplaying and generating operation 828 are, optionally, implemented byevent sorter 170, event recognizer 180, and event handler 190. Eventmonitor 171 in event sorter 170 detects a contact on touch-sensitivedisplay 112, and event dispatcher module 174 delivers the eventinformation to application 136-1. A respective event recognizer 180 ofapplication 136-1 compares the event information to respective eventdefinitions 186, and determines whether a first contact at a firstlocation on the touch-sensitive surface (or whether rotation of thedevice) corresponds to a predefined event or sub-event, such asselection of an object on a user interface, or rotation of the devicefrom one orientation to another. When a respective predefined event orsub-event is detected, event recognizer 180 activates an event handler190 associated with the detection of the event or sub-event. Eventhandler 190 optionally uses or calls data updater 176 or object updater177 to update the application internal state 192. In some embodiments,event handler 190 accesses a respective GUI updater 178 to update whatis displayed by the application. Similarly, it would be clear to aperson having ordinary skill in the art how other processes can beimplemented based on the components depicted in FIGS. 1A-1B.

FIGS. 10A-10D are flow diagrams illustrating method 1000 of settingpower/latency states of tactile output generators in accordance withsome embodiments. Method 1000 is performed at an electronic device(e.g., device 300, FIG. 3, or portable multifunction device 100, FIG.1A) with a display, a touch-sensitive surface, and one or more tactileoutput generators. In some embodiments, the display is a touch-screendisplay and the touch-sensitive surface is on or integrated with thedisplay. In some embodiments, the display is separate from thetouch-sensitive surface. Some operations in method 1000 are, optionally,combined and/or the order of some operations is, optionally, changed.

As described below, method 1000 provides an enhanced way to providetactile outputs based on states of tactile output generators. Predictingan upcoming tactile output generation event and transitioning tactileoutput generators into a low-latency mode reduces the latency ingenerating tactile outputs, which makes the device more responsive touser inputs, thereby creating a more efficient human-machine interface.For battery-operated electronic devices, placing the tactile outputgenerators in the low-latency mode based on prediction of upcomingtactile output generation events conserves power over maintaining thetactile output generators in the low-latency mode all the time andincreases the time between battery charges.

While the one or more tactile output generators are in a low-power state(e.g., at operation 5182, FIG. 5KKK), the device receives (1002) anindication that a user interaction has started (e.g., detecting a firstportion of an input such as a touch on a touch sensitive surface or thebeginning of a drag and drop operation as shown in FIG. 5III, orreceiving a “prepare” request from a first software application in whichthe user interaction has started, such as at operation 5184, FIG. 5KKK).

A respective portion of the user interaction is associated (1004) with atactile output (e.g., a portion of the user interaction that correspondsto an overlap of position indicator 5161 with marker 5162 is associatedwith a tactile output so that an overlap of position indicator 5161 andmarker 5162 initiates a tactile output).

The low-power state is (1006) a state in which tactile outputs aregenerated with a first amount of latency (e.g., it will take at least awarm-up amount of time, such as ΔT_(A), FIGS. 5ZZ and 5HHH to generatethe tactile output with the one or more tactile output generators).

In some embodiments, the low-power state is (1008) a predefined inactivestate of the one or more tactile output generators (e.g., “inactive”state, FIG. 5ZZ). In some embodiments, the one or more tactile outputgenerators consume no power in the low-power state.

In some embodiments, the user interaction that is associated with thetactile output includes (1010) location information corresponding to atouch input having a location on the touch-sensitive surface (e.g., alocation of position indicator 5161 in FIG. 5III) or a user interfaceobject having a location on the touch-sensitive surface.

In some embodiments, the device determines (1012) a location of thetactile output in accordance with the location information for the userinteraction (e.g., when the device includes multiple tactile outputgenerators, the tactile output is generated using one or more tactileoutput generators located adjacent to the location of the userinteraction).

In some embodiments, the one or more tactile output generators include(1014) two or more tactile output generators, each having acorresponding location, or location and size, and determining thelocation of the tactile output includes determining the location of thetactile output in accordance with the location information for the userinteraction and the locations, or locations and sizes, of the two ormore tactile output generators (e.g., as shown in FIGS. 5G-5H, thetactile output is generated at a location adjacent to a location ofslider thumb 507).

In response to receiving the indication that the user interaction thatis associated with the tactile output has started, in accordance with adetermination that the indication meets tactile output generatorpreparation criteria (e.g., at operation 5186-Yes, FIG. 5KKK), thedevice sets (1016) the one or more tactile output generators to alow-latency state (e.g., a pre-warmed state has a lower latency than aninactive state, and, in some embodiments, a higher latency than a readystate) at a first time (e.g., at T=t₁ in FIG. 5III). The low-latencystate is (1018) a state in which the tactile outputs are generated witha second amount of latency (e.g., ΔT_(B), FIGS. 5ZZ and 5III) that islower than the first amount of latency (e.g., the low-latency state is astate in which it will take less than the warm-up amount of time togenerate the tactile output with the one or more tactile outputgenerators as shown in FIG. 5ZZ).

In some embodiments, the low-power state of the one or more tactileoutput generators has (1020) a lower power consumption than thelow-latency state (e.g., the low-latency state is high-power state thatconsumes more power than the low-power state). The low-latency state hasa lower latency for generating tactile outputs than the low-power state,as shown in FIG. 5ZZ (e.g., the low-power state is high-latency statethat consumes less power than the low-latency state).

In some embodiments, the low-latency state is selected (1022) from afirst low-latency state (e.g., a pre-warmed state with a reducedlatency, such as a latency ΔT_(B) that is lower than a latency ΔT_(A) inthe off or inactive state but is still perceptible to a user, asdescribed herein with reference to FIG. 5ZZ) and a second low-latencystate (e.g., a ready or active state where the one or more tactileoutput generators are ready to generate tactile outputs with a minimallatency, such as a latency ΔT_(C) that is below the threshold at whichis perceptible to a user), as described for example with reference tooperation 5188 of FIG. 5KKK. The first low-latency state consumes lesspower than the second low-latency state and has a higher latency thanthe second low-latency state. For example, when a lowest latency isrequired, the device selects the second low-latency state.

In some embodiments, the low-latency state is selected (1024) from thefirst low-latency state and the second low-latency state by an operatingsystem of the device (e.g., operating system 126, FIG. 1A) based onenergy conservation protocols used by the device (e.g., if the firstsoftware application has been asking to set the device into the readystate too frequently, or the device is in a battery conservation mode ofoperation, the operating system can decide to set the device into thepre-warmed state instead of the ready state so as to conserve power).

In some embodiments, the device issues (1026) from a first softwareapplication a command to set the one or more tactile output generatorsto the low-latency state. Setting the one or more tactile outputgenerators to the low-latency state is performed in response to thecommand issued by the first software application to set the one or moretactile output generators to the low-latency state (e.g., information272 from application 1 (136-1) in FIG. 1D includes one or moreinstructions and/or commands to set the one or more tactile outputgenerators to the low-latency state).

In some embodiments, the device issues (1028) from a first softwareapplication (e.g., application 136-1, FIG. 1D) a command to set the oneor more tactile output generators to the low-latency state, and at anoperating system of the electronic device, receives the issued commandand makes a determination of whether to set the one or more tactileoutput generators to the low-latency state.

In some embodiments, the device determines (1030), using an operatingsystem of the electronic device, when to set the one or more tactileoutput generators to the low-latency state (e.g., operation 5186 in FIG.5KKK).

In some embodiments, the operating system of the electronic devicedetermines (1032) whether to set the one or more tactile outputgenerators to the low-latency state in accordance with a state of theelectronic device (e.g., if the device is in a battery conservation modeof operation, the operating system can decide whether to set the deviceinto the low-latency state or not, so as to conserve power).

In some embodiments, the operating system of the electronic devicedetermines (1034), in response to a first software application issuing acommand to set the one or more tactile output generators to thelow-latency state, whether to set the one or more tactile outputgenerators to the low-latency state in accordance with historicalinformation concerning prior commands by the first software applicationto set the one or more tactile output generators to the low-latencystate (e.g., if an application requests to prepare too many times,without requesting a tactile output to be generated, the device ignoressubsequent prepare requests).

After setting the one or more tactile output generators to thelow-latency state (1036), in accordance with a determination that theuser interaction has reached the respective portion of the userinteraction that is associated with the tactile output before apredefined amount of time since the first time has elapsed and that theone or more tactile output generators are still in the low-latencystate, the device generates (1038) the tactile output using the one ormore tactile output generators with the second amount of latency (e.g.,generating the tactile output takes less than the warm-up amount oftime); and, in accordance with a determination that a tactile output hasnot been generated for at least the predefined amount of time since thefirst time (e.g., a timeout period), transitions (1040) the one or moretactile output generators from the low-latency state to the low-powerstate (e.g., an inactive state) (e.g., using a UIFeedbackGeneratorprepare function described below in Appendix A), as described forexample with reference to operation 5194, FIG. 5KKK.

In some embodiments, after setting the one or more tactile outputgenerators to the low-latency state (1036), the device, in accordancewith a determination that the user interaction has reached therespective portion of the user interaction that is associated with thetactile output after the predefined amount of time since the first timehas elapsed and that the one or more tactile output generators havereturned to the low-power state, generates (1042) the tactile outputusing the one or more tactile output generators with the first amount oflatency (e.g., generating the tactile output takes at least the warm-upamount of time).

In some embodiments, in response to receiving the indication that theuser interaction that is associated with the tactile output has started,in accordance with a determination that the indication does not meettactile output generator preparation criteria (e.g., because the deviceis in a battery conservation mode of operation or because the firstapplication has been requesting that the device prepare to generatetactile outputs too frequently), the device maintains (1044) the one ormore tactile output generators in the low-power state and forgoessetting the one or more tactile output generators to the low-latencystate (e.g., if the first software application has been asking to setthe device into the ready state too frequently, or the device is in abattery conservation mode of operation, the operating system can ignorethe request in order to conserve battery power), as described forexample with reference to operation 5186-No, FIG. 5KKK.

It should be understood that the particular order in which theoperations in FIGS. 10A-10D have been described is merely an example andis not intended to indicate that the described order is the only orderin which the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein. Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,methods 600, 800, 1200, 1400, 1600, 1800, and 2000) are also applicablein an analogous manner to method 1000 described above with respect toFIGS. 10A-10D. For example, the tactile outputs, tactile outputgenerators, tactile output generator states, and software applicationsdescribed above with reference to method 1000 optionally have one ormore of the characteristics of the tactile outputs, tactile outputgenerators, tactile output generator states, and software applicationsdescribed herein with reference to other methods described herein (e.g.,methods 600, 800, 1200, 1400, 1600, 1800, and 2000). For brevity, thesedetails are not repeated here.

The operations in the information processing methods described aboveare, optionally implemented by running one or more functional modules ininformation processing apparatus such as general purpose processors(e.g., as described above with respect to FIGS. 1A and 3) or applicationspecific chips.

In accordance with some embodiments, FIG. 11 shows a functional blockdiagram of electronic device 1100 configured in accordance with theprinciples of the various described embodiments. The functional blocksof the device are, optionally, implemented by hardware, software, or acombination of hardware and software to carry out the principles of thevarious described embodiments. It is understood by persons of skill inthe art that the functional blocks described in FIG. 11 are, optionally,combined or separated into sub-blocks to implement the principles of thevarious described embodiments. Therefore, the description hereinoptionally supports any possible combination or separation or furtherdefinition of the functional blocks described herein.

As shown in FIG. 11, electronic device 1100 includes display unit 1102(e.g., including display 112) configured to display one or more userinterfaces, touch-sensitive surface unit 1104 configured to receivetouch inputs (e.g., on a display surface of display unit 1102), one ormore tactile output generator unit(s) 1106 configured to generate one ormore tactile outputs, and processing unit 1108 coupled with display unit1102, touch-sensitive surface unit 1104, and one or more tactile outputgenerator unit(s) 1106. In some embodiments, processing unit 1108includes one or more of the following sub-units: receiving unit 1110,state setting unit 1112, determining unit 1114, tactile output enablingunit 1116, and issuing unit 1118.

In some embodiments, while one or more tactile output generator units1106 are in a low-power state, processing unit 1108 is configured toreceive (e.g., using receiving unit 1110) an indication that a userinteraction has started, wherein a respective portion of the userinteraction is associated with a tactile output, and the low-power stateis a state in which tactile outputs are generated with a first amount oflatency. In response to receiving the indication that the userinteraction that is associated with the tactile output has started, inaccordance with a determination (e.g., made using determining unit 1114)that the indication meets tactile output generator preparation criteria,processing unit 1108 sets (e.g., using state setting unit 1112) one ormore tactile output generator units 1106 to a low-latency state at afirst time, wherein the low-latency state is a state in which thetactile outputs are generated with a second amount of latency that islower than the first amount of latency. After setting one or moretactile output generator units 1106 to the low-latency state, inaccordance with a determination (e.g., made using determining unit 1114)that the user interaction has reached the respective portion of the userinteraction that is associated with the tactile output before apredefined amount of time since the first time has elapsed and that oneor more tactile output generator units 1106 are still in the low-latencystate, processing unit 1108 enables generation of the tactile outputusing one or more tactile output generator units 1106 with the secondamount of latency. In accordance with a determination that a tactileoutput has not been generated for at least the predefined amount of timesince the first time, processing unit 1108 transitions one or moretactile output generator units 1106 from the low-latency state to thelow-power state.

In some embodiments, processing unit 1108 is further configured to,after setting one or more tactile output generator units 1106 to thelow-latency state, in accordance with a determination that the userinteraction has reached the respective portion of the user interactionthat is associated with the tactile output after the predefined amountof time since the first time has elapsed and that one or more tactileoutput generator units 1106 have returned to the low-power state, enablegeneration (e.g., using tactile output enabling unit 1116) of thetactile output using one or more tactile output generator units 1106with the first amount of latency.

In some embodiments, the low-power state of one or more tactile outputgenerator units 1106 has a lower power consumption than the low-latencystate, and the low-latency state has a lower latency for generatingtactile outputs than the low-power state.

In some embodiments, the low-power state is a predefined inactive stateof one or more tactile output generator units 1106.

In some embodiments, the low-latency state is selected from a firstlow-latency state and a second low-latency state, wherein the firstlow-latency state consumes less power than the second low-latency stateand has a higher latency than the second low-latency state.

In some embodiments, the low-latency state is selected from the firstlow-latency state and the second low-latency state by an operatingsystem of device 1100 based on energy conservation protocols used bydevice 1100.

In some embodiments, processing unit 1108 is further configured to, inresponse to receiving the indication (e.g., using receiving unit 1110)that the user interaction that is associated with the tactile output hasstarted, in accordance with a determination that the indication does notmeet tactile output generator preparation criteria, maintain (e.g.,using state setting unit 1112) one or more tactile output generatorunits 1106 in the low-power state and forgo setting one or more tactileoutput generator units 1106 to the low-latency state.

In some embodiments, the user interaction that is associated with thetactile output includes location information corresponding to a touchinput having a location on touch-sensitive surface unit 1104 or a userinterface object having a location on touch-sensitive surface unit 1104.

In some embodiments, processing unit 1108 is further configured todetermine a location (e.g., using determining unit 1114) of the tactileoutput in accordance with the location information for the userinteraction.

In some embodiments, one or more tactile output generator units 1106include two or more tactile output generator units 1106, each having acorresponding location, or location and size, and determining thelocation of the tactile output includes determining (e.g., usingdetermining unit 1114) the location of the tactile output in accordancewith the location information for the user interaction and thelocations, or locations and sizes, of two or more tactile outputgenerator units 1106.

In some embodiments, processing unit 1108 is further configured to issue(e.g., using issuing unit 1118) from a first software application acommand to set (e.g., using state setting unit 1112) one or more tactileoutput generator units 1106 to the low-latency state, wherein settingone or more tactile output generator units 1106 to the low-latency stateis performed in response to the command issued by the first softwareapplication to set one or more tactile output generator units 1106 tothe low-latency state.

In some embodiments, processing unit 1108 is further configured to issue(e.g., using issuing unit 1118) a command to set (e.g., using statesetting unit 1112) one or more tactile output generator units 1106 tothe low-latency state, and receive, at an operating system of electronicdevice 1100, the issued command and making a determination (e.g., usingdetermining unit 1114) of whether to set one or more tactile outputgenerator units 1106 to the low-latency state.

In some embodiments, processing unit 1108 is further configured todetermine (e.g., using determining unit 1114), using an operating systemof electronic device 1100, when to set (e.g., using state setting unit1112) one or more tactile output generator units 1106 to the low-latencystate. In some embodiments, processing unit 1108 is further configuredto determine (e.g., using determining unit 1114), using the operatingsystem of electronic device 1100, whether to set (e.g., using statesetting unit 1112) one or more tactile output generator units 1106 tothe low-latency state in accordance with a state of electronic device1100.

In some embodiments, the operating system of electronic device 1100determines (e.g., using determining unit 1114), in response to a firstsoftware application issuing a command to set one or more tactile outputgenerator units 1106 to the low-latency state, whether to set one ormore tactile output generator units 1106 to the low-latency state inaccordance with historical information concerning prior commands by thefirst software application to set one or more tactile output generatorunits 1106 to the low-latency state.

The operations described above with reference to FIGS. 10A-10D are,optionally, implemented by components depicted in FIGS. 1A-1B or FIG.11. For example, receiving operation 1002, setting operation 1016, andgenerating operation 1038 are, optionally, implemented by event sorter170, event recognizer 180, and event handler 190. Event monitor 171 inevent sorter 170 detects a contact on touch-sensitive display 112, andevent dispatcher module 174 delivers the event information toapplication 136-1. A respective event recognizer 180 of application136-1 compares the event information to respective event definitions186, and determines whether a first contact at a first location on thetouch-sensitive surface (or whether rotation of the device) correspondsto a predefined event or sub-event, such as selection of an object on auser interface, or rotation of the device from one orientation toanother. When a respective predefined event or sub-event is detected,event recognizer 180 activates an event handler 190 associated with thedetection of the event or sub-event. Event handler 190 optionally usesor calls data updater 176 or object updater 177 to update theapplication internal state 192. In some embodiments, event handler 190accesses a respective GUI updater 178 to update what is displayed by theapplication. Similarly, it would be clear to a person having ordinaryskill in the art how other processes can be implemented based on thecomponents depicted in FIGS. 1A-1B.

FIGS. 12A-12D are flow diagrams illustrating method 1200 of generatingconsistent tactile outputs for multiple applications in accordance withsome embodiments. Method 1200 is performed (1202) at an electronicdevice (e.g., device 300, FIG. 3, or portable multifunction device 100,FIG. 1A) with a display, a touch-sensitive surface, and one or moretactile output generators. In some embodiments, the display is atouch-screen display and the touch-sensitive surface is on or integratedwith the display. In some embodiments, the display is separate from thetouch-sensitive surface. Some operations in method 1200 are, optionally,combined and/or the order of some operations is, optionally, changed.

As described below, method 1200 provides an enhanced way to providetactile outputs using an application-independent software module.Providing tactile outputs using application-independent software basedon information from application software provides common user interfaceframework that provides consistent user experience when various softwareapplications are used. Providing a common user interface frame work tothe user enhances the usability of such software applications and thedevice executing such software applications. In turn, this enhances theoperability of the device and makes the user-device interface moreefficient (e.g., by helping the user to provide proper inputs andreducing user mistakes and/or unintended operations whenoperating/interacting with the device). In addition, the method reducesthe size of software applications and makes execution of such softwareapplications faster. For battery-operated electronic devices, enabling auser to use software applications faster and more efficiently conservespower and increases the time between battery charges.

While a first type of event is associated with a first tactile outputand a second type of event that is distinct from the first type of eventis associated with a second tactile output that is different from thefirst tactile output by an application-independent module (e.g.,application-independent software module 260, FIG. 1D), the devicereceives (1204) first information (e.g., information 272 indicatingwhether a particular operation corresponds to a success event or afailure event, FIG. 1D) from a first application (e.g., application136-1, FIG. 1D); and, in response to receiving the first informationfrom the first application (1206), generates a response to the firstinformation, including: in accordance with a determination that thefirst information corresponds to an application event of the first type(e.g., a success event), generating (1208) the first tactile outputusing the one or more tactile output generators (e.g., a series oftactile outputs for a success event illustrated in FIG. 5AAA); and, inaccordance with a determination that the first information correspondsto an application event of the second type (e.g., a failure event) thatis distinct from the first type, generating (1210), using the one ormore tactile output generators, the second tactile output that isdifferent from the first tactile output (e.g., a series of tactileoutputs for a failure event illustrated in FIG. 5BBB).

After generating the response to the first information, the devicereceives (1212) second information from a second application (e.g.,application 136-2, FIG. 1D) that is distinct from the first application;and, in response to receiving the second information from the secondapplication, generates (1214) a response to the second information,including: in accordance with a determination that the secondinformation corresponds to an application event of the first type,generating (1216) the first tactile output using the one or more tactileoutput generators (e.g., a series of tactile outputs for a success eventfor an authorization of an electronic payment transaction); and, inaccordance with a determination that the second information correspondsto an application event of the second type, generating (1218) the secondtactile output using the one or more tactile output generators (e.g., aseries of tactile outputs for a failure event for a denial of anelectronic payment transaction) (e.g., using aUINotificationFeedbackGenerator object (or class), including typesUINotificationFeedbackTypeSuccess and UINotificationFeedbackTypeError,as described in Appendix A).

In some embodiments, the first type of event is (1220) a success event(e.g., an event with the UIEventFeedbackTypeSuccess type, as describedin Appendix A) and the second type of event is a failure event (e.g., anevent with the UIEventFeedbackTypeError type, as described in AppendixA).

In some embodiments, the first information is received (1222) from anapplication-specific module for the first application (e.g., application136-1, or a component of application 136-1 such as application core270-1, FIG. 1E), the second information is received from anapplication-specific module for the second application (e.g.,application 136-2, or a component of application 136-2 such asapplication core 270-2, FIG. 1E), and determining whether to generatethe first tactile output or the second tactile output is performed bythe application-independent module (e.g., application-independentsoftware module 260, FIG. 1D, or a respective instance ofapplication-independent software module 260, FIG. 1E) (e.g., in FIG. 1E,application core 1 (270-1) sends information indicating whether acorresponding event is a success event or a failure event, applicationcore 2 (270-2) sends information indicating whether a correspondingevent is a success event or a failure event, and application independentsoftware module 260 (or its instances) determines whether to generatethe first tactile output or the second tactile output based onassociation of a success event and a failure event with the firsttactile output and the second tactile output).

In some embodiments, generating the response to the first informationincludes (1224), in accordance with a determination that the firstinformation corresponds to an application event of a third type (e.g., awarning event, such as a notification corresponding to display of anundo operation affordance as illustrated in FIG. 5CCC) that isassociated with a fifth tactile output that is different from the firsttactile output and the second tactile output and that is distinct fromthe first type and second type, generating, using the one or moretactile output generators, the fifth tactile output that is differentfrom the first tactile output and the second tactile output; andgenerating the response to the second information includes, inaccordance with a determination that the second information correspondsto an application event of the third type, generating the fifth tactileoutput using the one or more tactile output generators (e.g., using aUIEventFeedbackGenerator object (or class), including typeUIEventFeedbackTypeWarning, as described in Appendix A), as describedfor example with reference to FIG. 5CCC.

In some embodiments, after generating the response to the firstinformation and generating the response to the second information, thedevice changes (1226) the first tactile output associated with the firsttype of event by the application-independent module, includingassociating the first type of event with a third tactile output that isdifferent from the first tactile output. For example, because theapplication-independent module determines which tactile output togenerate for a respective type of event, by changing the association ofan event type with a particular tactile output in theapplication-independent module, tactile outputs for multipleapplications can be updated consistently. In addition, this facilitateschanging tactile outputs for a particular event type as the changes needto be made only with the application-independent module instead ofchanging all of the applications that are configured to generate tactileoutputs for the particular event type.

In some embodiments, after changing the tactile output associated withthe first type of event by the application-independent module, thedevice receives (1228) third information from the first application;and, in response to receiving the third information from the firstapplication, generating a response to the third information, including:in accordance with a determination that the third informationcorresponds to an application event of the first type (e.g., a successevent), generating the third tactile output using the one or moretactile output generators. After generating the response to the thirdinformation, the device receives fourth information from the secondapplication that is distinct from the first application; and, inresponse to receiving the fourth information from the secondapplication, generating a response to the fourth information, including,in accordance with a determination that the fourth informationcorresponds to an application event of the first type (e.g., a successevent), generating the third tactile output using the one or moretactile output generators. For example, once the application event ofthe first type (e.g., a success event) is associated with a new tactileoutput, namely the third tactile output, tactile outputs for theapplication event of the first type in both the first application andthe second application are changed to match the third tactile output.

In some embodiments, the third information is received (1230) from anapplication-specific module for the first application (e.g., applicationcore 1 (270-1) in FIG. 1E), the fourth information is received from anapplication-specific module for the second application (e.g.,application core 2 (270-2) in FIG. 1E), and determining whether togenerate the third tactile output (e.g., determining whether the thirdinformation and/or the fourth information correspond to an applicationevent of the first type) is performed by the application-independentmodule (e.g., application independent software module 260 or itsinstances in FIG. 1E).

In some embodiments, after generating the response to the firstinformation and generating the response to the second information, thedevice changes (1232) the tactile output associated with the second typeof event by the application-independent module, including associatingthe second type of event with a fourth tactile output that is differentfrom the second tactile output.

In some embodiments, after changing the tactile output associated withthe second type of event by the application-independent module, thedevice receives (1234) fifth information from the first application;and, in response to receiving the fifth information from the firstapplication, generates a response to the fifth information, including,in accordance with a determination that the fifth informationcorresponds to an application event of the second type (e.g., a failureevent), generating the fourth tactile output using the one or moretactile output generators. After generating the response to the fifthinformation, the device receives sixth information from the secondapplication that is distinct from the first application; and, inresponse to receiving the sixth information from the second application,generates a response to the sixth information, including, in accordancewith a determination that the sixth information corresponds to anapplication event of the second type, generating the fourth tactileoutput using the one or more tactile output generators. For example,once the application event of the second type (e.g., a failure event) isassociated with a new tactile output, namely the fourth tactile output,tactile outputs for the application event of the second type in both thefirst application and the second application are changed to match thefourth tactile output.

In some embodiments, the fifth information is received (1236) from anapplication-specific module for the first application (e.g., applicationcore 1 (270-1) in FIG. 1E), the sixth information is received from anapplication-specific module for the second application (e.g.,application core 2 (270-2) in FIG. 1E), and determining whether togenerate the fourth tactile output (e.g., determining whether the fifthinformation and/or the sixth information correspond to an applicationevent of the second type) is performed by the application-independentmodule (e.g., application independent software module 260 or itsinstances in FIG. 1E).

It should be understood that the particular order in which theoperations in FIGS. 12A-12D have been described is merely an example andis not intended to indicate that the described order is the only orderin which the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein. Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,methods 600, 800, 1000, 1400, 1600, 1800, and 2000) are also applicablein an analogous manner to method 1200 described above with respect toFIGS. 12A-12D. For example, the tactile outputs, information, softwareapplications, application-independent modules, and application-specificmodules described above with reference to method 1200 optionally haveone or more of the characteristics of the tactile outputs, information,software applications, application-independent modules, andapplication-specific modules described herein with reference to othermethods described herein (e.g., methods 600, 800, 1000, 1400, 1600,1800, and 2000). For brevity, these details are not repeated here.

The operations in the information processing methods described aboveare, optionally implemented by running one or more functional modules ininformation processing apparatus such as general purpose processors(e.g., as described above with respect to FIGS. 1A and 3) or applicationspecific chips.

The operations described above with reference to FIGS. 12A-12D are,optionally, implemented by components depicted in FIGS. 1A-1B or FIG.13. For example, receiving operation 1204, generating operation 1208,receiving operation 1212, and generating operation 1214 are, optionally,implemented by event sorter 170, event recognizer 180, and event handler190. Event monitor 171 in event sorter 170 detects a contact ontouch-sensitive display 112, and event dispatcher module 174 deliversthe event information to application 136-1. A respective eventrecognizer 180 of application 136-1 compares the event information torespective event definitions 186, and determines whether a first contactat a first location on the touch-sensitive surface (or whether rotationof the device) corresponds to a predefined event or sub-event, such asselection of an object on a user interface, or rotation of the devicefrom one orientation to another. When a respective predefined event orsub-event is detected, event recognizer 180 activates an event handler190 associated with the detection of the event or sub-event. Eventhandler 190 optionally uses or calls data updater 176 or object updater177 to update the application internal state 192. In some embodiments,event handler 190 accesses a respective GUI updater 178 to update whatis displayed by the application. Similarly, it would be clear to aperson having ordinary skill in the art how other processes can beimplemented based on the components depicted in FIGS. 1A-1B.

In accordance with some embodiments, FIG. 13 shows a functional blockdiagram of electronic device 1300 configured in accordance with theprinciples of the various described embodiments. The functional blocksof the device are, optionally, implemented by hardware, software, or acombination of hardware and software to carry out the principles of thevarious described embodiments. It is understood by persons of skill inthe art that the functional blocks described in FIG. 13 are, optionally,combined or separated into sub-blocks to implement the principles of thevarious described embodiments. Therefore, the description hereinoptionally supports any possible combination or separation or furtherdefinition of the functional blocks described herein.

As shown in FIG. 13, electronic device 1300 includes display unit 1302(e.g., including display 112) configured to display user interfaces,touch-sensitive surface unit 1304 configured to receive touch inputs(e.g., on a display surface of display unit 1302), one or more tactileoutput generator unit(s) 1306 configured to generate one or more tactileoutputs, and processing unit 1308 coupled with display unit 1302,touch-sensitive surface unit 1304, and one or more tactile outputgenerator unit(s) 1306. In some embodiments, processing unit 1308includes one or more of the following sub-units: receiving unit 1310,response generation enabling unit 1312, determining unit 1314, andchanging unit 1316.

In some embodiments, while a first type of event is associated with afirst tactile output and a second type of event that is distinct fromthe first type is associated with a second tactile output that isdifferent from the first tactile output by an application-independentmodule, processing unit 1308 is configured to receive (e.g., usingreceiving unit 1310) first information from a first application.Processing unit 1308 is configured to, in response to receiving thefirst information from the first application, enable generation of aresponse (e.g., using response generation enabling unit 1312) to thefirst information, including, in accordance with a determination (e.g.,made using determining unit 1314) that the first information correspondsto an application event of the first type, enabling generation of thefirst tactile output (e.g., using response generation enabling unit1312) using one or more tactile output generator units 1306, and inaccordance with a determination (e.g., made using determining unit 1314)that the first information corresponds to an application event of thesecond type that is distinct from the first type, enabling generation(e.g., using response generation enabling unit 1312), using one or moretactile output generator units 1306, of the second tactile output thatis different from the first tactile output. Processing unit 1308 isconfigured to, after enabling generation of the response to the firstinformation, receive (e.g., using receiving unit 1310) secondinformation from a second application that is distinct from the firstapplication. Processing unit 1308 is configured to, in response toreceiving the second information from the second application, enablegeneration of a response (e.g., using response generation enabling unit1312) to the second information, including, in accordance with adetermination (e.g., made using determining unit 1314) that the secondinformation corresponds to an application event of the first type,enabling generation of the first tactile output using one or moretactile output generator units 1306, and in accordance with adetermination (e.g., made using determining unit 1314) that the secondinformation corresponds to an application event of the second type,enabling generation of the second tactile output using one or moretactile output generator units 1306.

In some embodiments, processing unit 1308 is further configured to,after enabling generation of the response to the first information andenabling generation of the response to the second information, changethe tactile output (e.g., using changing unit 1316) associated with thefirst type of event by the application-independent module, includingassociating the first type of event with a third tactile output that isdifferent from the first tactile output.

In some embodiments, after changing the tactile output associated withthe first type of event by the application-independent module,processing unit 1308 receives third information (e.g., using receivingunit 1310) from the first application. Processing unit 1308 isconfigured to, in response to receiving the third information from thefirst application, enable generation of a response (e.g., using responsegeneration enabling unit 1312) to the third information, including, inaccordance with a determination (e.g., made using determining unit 1314)that the third information corresponds to an application event of thefirst type, enabling generation of the third tactile output (e.g., usingresponse generation enabling unit 1312) using one or more tactile outputgenerator units 1306. Processing unit 1308 is configured to, afterenabling generation of the response to the third information, receivefourth information (e.g., using receiving unit 1310) from the secondapplication that is distinct from the first application. Processing unit1308 is configured to, in response to receiving the fourth informationfrom the second application, enable generation of a response (e.g.,using response generation enabling unit 1312) to the fourth information,including, in accordance with a determination (e.g., made usingdetermining unit 1314) that the fourth information corresponds to anapplication event of the first type, enabling generation of the thirdtactile output (e.g., using response generation enabling unit 1312)using one or more tactile output generator units 1306.

In some embodiments, the third information is received from anapplication-specific module for the first application, the fourthinformation is received from an application-specific module for thesecond application, and determining whether to generate the thirdtactile output is performed by the application-independent module.

In some embodiments, processing unit 1308 is configured to, afterenabling generation of the response to the first information andenabling generation of the response to the second information, change(e.g., using changing unit 1316) the tactile output associated with thesecond type of event by the application-independent module, includingassociating the second type of event with a fourth tactile output thatis different from the second tactile output.

In some embodiments, processing unit 1308 is configured to, afterchanging (e.g., using changing unit 1316) the tactile output associatedwith the second type of event by the application-independent module,receive fifth information (e.g., using receiving unit 1310) from thefirst application. Processing unit 1308 is configured to, in response toreceiving the fifth information from the first application, enablegeneration of a response (e.g., using response generation enabling unit1312) to the fifth information, including, in accordance with adetermination (e.g., made using determining unit 1314) that the fifthinformation corresponds to an application event of the second type,enabling generation of the fourth tactile output (e.g., using responsegeneration enabling unit 1312) using one or more tactile outputgenerator units 1306. Processing unit 1308 is configured to, afterenabling generation of the response to the fifth information, receivesixth information (e.g., using receiving unit 1310) from the secondapplication that is distinct from the first application. Processing unit1308 is configured to, in response to receiving the sixth informationfrom the second application, enable generation of a response (e.g.,using response generation enabling unit 1312) to the sixth information,including, in accordance with a determination (e.g., made usingdetermining unit 1314) that the sixth information corresponds to anapplication event of the second type, enabling generation of the fourthtactile output (e.g., using response generation enabling unit 1312)using one or more tactile output generator units 1306.

In some embodiments, the fifth information is received from anapplication-specific module for the first application, the sixthinformation is received from an application-specific module for thesecond application, and determining whether to generate the fourthtactile output is performed by the application-independent module.

In some embodiments, the first information is received from anapplication-specific module for the first application, the secondinformation is received from an application-specific module for thesecond application, and determining whether to generate the firsttactile output or the second tactile output is performed by theapplication-independent module.

In some embodiments, enabling generation of the response (e.g., usingresponse generation enabling unit 1312) to the first informationincludes, in accordance with a determination (e.g., made usingdetermining unit 1314) that the first information corresponds to anapplication event of a third type that is associated with a fifthtactile output that is different from the first tactile output and thesecond tactile output and that is distinct from the first type andsecond type, enabling generation (e.g., using response generationenabling unit 1312), using one or more tactile output generator units1306, of the fifth tactile output that is different from the firsttactile output and the second tactile output. In some embodiments,enabling generation of the response to the second information includes,in accordance with a determination (e.g., made using determining unit1314) that the second information corresponds to an application event ofthe third type, enabling generation (e.g., using response generationenabling unit 1312) of the fifth tactile output using one or moretactile output generator units 1306.

In some embodiments, the first type of event is a success event and thesecond type of event is a failure event.

FIGS. 14A-14B are flow diagrams illustrating method 1400 of generatingtactile outputs based on information from an application-specific modulein accordance with some embodiments. Method 1400 is performed at anelectronic device (e.g., device 300, FIG. 3, or portable multifunctiondevice 100, FIG. 1A) with a display, a touch-sensitive surface, and oneor more sensors to detect intensity of contacts with the touch-sensitivesurface. In some embodiments, the display is a touch-screen display andthe touch-sensitive surface is on or integrated with the display. Insome embodiments, the display is separate from the touch-sensitivesurface. Some operations in method 1400 are, optionally, combined and/orthe order of some operations is, optionally, changed.

As described below, method 1400 provides an enhanced way to providetactile outputs using an application-independent software module.Providing tactile outputs using application-independent software basedon information from application software provides common user interfaceframework that provides consistent user experience when various softwareapplications are used. Providing a common user interface frame work tothe user enhances the usability of such software applications and thedevice executing such software applications. In addition, the methodprovides an intuitive way to provide tactile outputs based on physicssimulation of impact events. In turn, these features enhance theoperability of the device and make the user-device interface moreefficient (e.g., by helping the user to provide proper inputs andreducing user mistakes and/or unintended operations whenoperating/interacting with the device). Furthermore, the method reducesthe size of software applications and makes execution of such softwareapplications faster. For battery-operated electronic devices, enabling auser to use software applications faster and more efficiently conservespower and increases the time between battery charges.

The device receives (1402), at an application-independent module (e.g.,application-independent software module 260, FIG. 1D), from anapplication-specific module that is associated with a first application(e.g., application 136-1, or a component of application 136-1 such asapplication core 270-1, FIGS. 1D-1E), information (e.g., information272) about an input directed to a user interface of the firstapplication.

An operation performed in the user interface of the first application inresponse to detecting the input is (1404) associated with a tactileoutput pattern specified by the application-specific module.

In addition, the information about the input includes (1406) informationindicating a magnitude (e.g., speed or distance) of the operationperformed in the user interface in response to detecting the input, asdescribed in further detail herein with reference to FIGS. 5DDD-5GGG.

In some embodiments, the magnitude is (1408) provided by the firstapplication.

In some embodiments, the magnitude is (1410) provided by an operatingsystem of the electronic device (e.g., operating system 126, FIGS. 1Aand 3). In some embodiments, the magnitude provided by the operatingsystem takes into account other tactile outputs being generated duringan overlapping time period.

In response to receiving the information about the input directed to theuser interface of the first application, the device generates (1412),via the one or more tactile output generators, a tactile output thatcorresponds to the operation performed in the user interface of thefirst application.

The tactile output has (1414) the tactile output pattern specified bythe application-specific module.

In addition, the tactile output has (1416) an amplitude determined(e.g., by the application-independent module) in accordance with themagnitude of the operation performed in the user interface of the firstapplication in response to detecting the input (e.g., as provided by theapplication-specific module), as described in further detail herein withreference to FIGS. 5DDD-5GGG. For example, the tactile output isgenerated using UIImpactFeedbackGenerator.

In some embodiments, the amplitude of the tactile output is (1418)further determined in accordance with at least one of a velocity or massof an object in a simulated collision or impact.

In some embodiments, the amplitude of the tactile output is (1420)further determined by applying a predefined physics model to one or moreparameters of one or more displayed user interface elements or one ormore parameters of a touch input.

In some embodiments, the tactile output is (1422) further characterizedby a vibration frequency, wherein the vibration frequency is selectedbased on the tactile output pattern.

In some embodiments, the vibration frequency is (1424) less than 100hertz.

It should be understood that the particular order in which theoperations in FIGS. 14A-14B have been described is merely an example andis not intended to indicate that the described order is the only orderin which the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein. Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,methods 600, 800, 1000, 1200, 1600, 1800, and 2000) are also applicablein an analogous manner to method 1400 described above with respect toFIGS. 14A-14B. For example, the tactile outputs, user interfaceoperations, magnitudes, amplitudes, and frequencies described above withreference to method 1400 optionally have one or more of thecharacteristics of the tactile outputs, user interface operations,magnitudes, amplitudes, and frequencies described herein with referenceto other methods described herein (e.g., methods 600, 800, 1000, 1200,1600, 1800, and 2000). For brevity, these details are not repeated here.

The operations in the information processing methods described aboveare, optionally implemented by running one or more functional modules ininformation processing apparatus such as general purpose processors(e.g., as described above with respect to FIGS. 1A and 3) or applicationspecific chips.

In accordance with some embodiments, FIG. 15 shows a functional blockdiagram of electronic device 1500 configured in accordance with theprinciples of the various described embodiments. The functional blocksof the device are, optionally, implemented by hardware, software, or acombination of hardware and software to carry out the principles of thevarious described embodiments. It is understood by persons of skill inthe art that the functional blocks described in FIG. 15 are, optionally,combined or separated into sub-blocks to implement the principles of thevarious described embodiments. Therefore, the description hereinoptionally supports any possible combination or separation or furtherdefinition of the functional blocks described herein.

As shown in FIG. 15, electronic device 1500 includes display unit 1502(e.g., including display 112) configured to display user interfaces,touch-sensitive surface unit 1504 configured to receive touch inputs(e.g., on a surface, such as a display surface of display unit 1502),one or more tactile output generator unit(s) 1506 configured to generateone or more tactile outputs, and processing unit 1508 coupled withdisplay unit 1502, touch-sensitive surface unit 1504, and one or moretactile output generator unit(s) 1506. In some embodiments, processingunit 1508 includes one or more of the following sub-units: receivingunit 1510, tactile output enabling unit 1512, and determining unit 1514.

In some embodiments, processing unit 1508 is configured to receive(e.g., using receiving unit 1510), at an application-independent module,from an application-specific module that is associated with a firstapplication, information about an input directed to the user interfaceof the first application, wherein an operation performed in the userinterface of the first application in response to detecting the input isassociated with a tactile output pattern specified by theapplication-specific module, and the information about the inputincludes information indicating a magnitude of the operation performedin the user interface in response to detecting the input. Processingunit 1508 is configured to, in response to receiving the informationabout the input directed to the user interface of the first application,enable generation (e.g., using tactile output enabling unit 1512), viaone or more tactile output generator units 1506, of a tactile outputthat corresponds to the operation performed in the user interface of thefirst application, wherein the tactile output has the tactile outputpattern specified by the application-specific module, and the tactileoutput has an amplitude determined in accordance with the magnitude ofthe operation performed in the user interface of the first applicationin response to detecting the input.

In some embodiments, the tactile output is further characterized by avibration frequency, wherein the vibration frequency is selected basedon the tactile output pattern. In some embodiments, the vibrationfrequency is less than 100 hertz. In some embodiments, the amplitude ofthe tactile output is further determined (e.g., using determining unit1514) in accordance with at least one of a velocity or mass of an objectin a simulated collision or impact. In some embodiments, the amplitudeof the tactile output is further determined (e.g., using determiningunit 1514) by applying a predefined physics model to one or moreparameters of one or more displayed user interface elements or one ormore parameters of a touch input. In some embodiments, the magnitude isprovided by the application. In some embodiments, the magnitude isprovided by an operating system of electronic device 1500.

The operations described above with reference to FIGS. 14A-14B are,optionally, implemented by components depicted in FIGS. 1A-1B or FIG.15. For example, receiving operation 1402 and generating operation 1412are, optionally, implemented by event sorter 170, event recognizer 180,and event handler 190. Event monitor 171 in event sorter 170 detects acontact on touch-sensitive display 112, and event dispatcher module 174delivers the event information to application 136-1. A respective eventrecognizer 180 of application 136-1 compares the event information torespective event definitions 186, and determines whether a first contactat a first location on the touch-sensitive surface (or whether rotationof the device) corresponds to a predefined event or sub-event, such asselection of an object on a user interface, or rotation of the devicefrom one orientation to another. When a respective predefined event orsub-event is detected, event recognizer 180 activates an event handler190 associated with the detection of the event or sub-event. Eventhandler 190 optionally uses or calls data updater 176 or object updater177 to update the application internal state 192. In some embodiments,event handler 190 accesses a respective GUI updater 178 to update whatis displayed by the application. Similarly, it would be clear to aperson having ordinary skill in the art how other processes can beimplemented based on the components depicted in FIGS. 1A-1B.

FIGS. 16A-16D are flow diagrams illustrating method 1600 of settingpower/latency states of tactile output generators based on userinteraction events in accordance with some embodiments. Method 1600 isperformed at an electronic device (e.g., device 300, FIG. 3, or portablemultifunction device 100, FIG. 1A) with a display, a touch-sensitivesurface, and one or more sensors to detect intensity of contacts withthe touch-sensitive surface. In some embodiments, the display is atouch-screen display and the touch-sensitive surface is on or integratedwith the display. In some embodiments, the display is separate from thetouch-sensitive surface. Some operations in method 1600 are, optionally,combined and/or the order of some operations is, optionally, changed.

As described below, method 1600 provides an enhanced way to providetactile outputs based on states of tactile output generators. Predictingan upcoming tactile output generation event and transitioning tactileoutput generators into a low-latency mode reduces the latency ingenerating tactile outputs, which makes the device more responsive touser inputs, thereby creating a more efficient human-machine interface.For battery-operated electronic devices, placing the tactile outputgenerators in the low-latency mode based on prediction of upcomingtactile output generation events conserves power over maintaining thetactile output generators in the low-latency mode all the time andincreases the time between battery charges.

The device displays (1602) a user interface on the display.

While displaying the user interface on the display and while the one ormore tactile output generators are in a low-power state (e.g., inactive,FIGS. 5ZZ and 5HHH), the device detects (1604) a first user interactionvia the touch-sensitive surface.

In some embodiments, the first user interaction is (1606) touchdown of acontact on the touch-sensitive surface.

In some embodiments, the first user interaction is (1608) a change in acontact after detecting touchdown of the contact on the touch-sensitivesurface.

In response to detecting the first user interaction, the device sets(1610) the one or more tactile output generators to a low-latency state(e.g., pre-warm or active, FIG. 5ZZ), as described in further detailherein with reference to FIG. 5III. In some embodiments, the tactileoutput generators are set to the low-latency state based on adetermination that the first user interaction is one that potentiallycorresponds to a sequence of input events (e.g., a drag and drop input)that are associated with tactile outputs. For example, in someembodiments the tactile output generators are set to the low-latencystate as described herein with reference to method 1000.

After setting the one or more tactile output generators to thelow-latency state, the device detects (1612) a second user interactionthat is part of a same sequence of user interactions as the first userinteraction (e.g., the first user interaction and the second userinteraction correspond to a same continuously detected contact or set ofcontacts on the touch-sensitive surface).

In response to detecting the second user interaction, the devicegenerates (1614) a tactile output that corresponds to the second userinteraction (e.g., using a UIFeedbackGenerator Prepare function or aUIFeedbackGeneratorUserInteractionDriven option as described in AppendixB).

Referring again to operation 1612, in some embodiments, the first userinteraction and the second user interaction are (1616) detected as partof a continuous touch input on the touch-sensitive surface (e.g., duringwhich the touch input remains in contact with the touch-sensitivesurface) that traverses a sequence of components of the user interfacedisplayed on the display.

In some embodiments, the device scrolls (1618) a set of menu items inresponse to user input on the touch-sensitive surface, wherein thesequence of user interactions comprises a sequence of respective menuitems in the set of menu items scrolling to or through a selectionposition in the user interface (e.g., as described in further detailherein with reference to FIGS. 5I-5J).

In some embodiments, the device scrolls (1620) a set of date componentor time component items in response to user input on the touch-sensitivesurface, wherein the sequence of user interactions comprises a sequenceof date component or time component items scrolling to or through aselection position in the user interface (e.g., as described in furtherdetail herein with reference to FIGS. 5K-5O).

Referring again to operation 1614, in some embodiments, a monitoringfunction tracks (1622) a progress of the sequence of user interactions.The one or more tactile output generators are changed to the low-latencystate based on a determination that the monitoring function has entereda start state in which the monitoring function has started monitoringthe sequence of user interactions. In addition, the tactile output isgenerated based on a determination that the monitoring function hasentered a changed state (e.g., a changed state for or with respect tothe sequence of user interactions).

In some embodiments, a monitoring function tracks (1624) a progress ofthe sequence of user interactions. The one or more tactile outputgenerators are changed to the low-latency state based on a determinationthat the monitoring function has entered a changed state that precedes achanged state that corresponds to a tactile output (e.g., a changedstate indicating movement of the contact). In addition, the tactileoutput is generated based on a determination that the monitoringfunction has entered the changed state that corresponds to a tactileoutput (e.g., a changed state indicating movement of the contact over aboundary or to a new drop target).

In some embodiments, the sequence of user interactions corresponds(1626) to a continuous touch input on the touch-sensitive surface thattraverses a sequence of components of a user interface displayed on thedisplay, and the device generates a sequence of two or more tactileoutputs using identical waveforms (e.g., usingUIRetargetFeedbackGenerator).

In some embodiments, the sequence of user interactions corresponds(1628) to a touch input on the touch-sensitive surface that traverses asequence of soft keys (e.g., keys for selecting different versions of asame letter or character, such as a letter (e.g., “e”) with differentdiacritical marks) displayed on the display, and the device generatesone or more tactile outputs that correspond to the touch input reachingpredefined positions with respect to the sequence of soft keys (e.g.,leading edges, or central positions, or both, of the keys).

In some embodiments, the sequence of user interactions includes (1630) asequence of events corresponding to distinct phases of dragging anobject, in the user interface displayed on the display, from an initiallocation to another location (e.g., using a UIDragFeedbackGeneratorobject), as described in further detail herein with reference to FIGS.5LL-5QQ.

In some embodiments, the distinct phases of dragging the object include(1632) an object lifting phase and an object dropping phase. In someembodiments, in response to a first respective user interaction, in thesequence of user interactions, the first respective user interactioncorresponding to the object lifting phase, the device sets one or moretactile output generators to the low-latency state, and in response to asecond respective user interaction, in the sequence of userinteractions, the second respective user interaction corresponding tothe object dropping phase, the device sets one or more tactile outputgenerators to the low-power state (e.g., as described herein withreference to method 1000).

In some embodiments, the distinct phases of dragging the object include(1634) an object snapping phase, corresponding to the object snappinginto place with respect to a user interface feature (e.g., a grid or anarray, as described in further detail herein with reference to FIGS.5RR-5SS), and the tactile output corresponds to the object snapping intoplace with respect to the user interface feature. In some embodiments, atactile output is generated in accordance with an object snapping intoplace with respect to a user interface feature without regard to whetherthe object snapping into place corresponds to an object snapping phaseof a sequence of distinct phases of dragging the object.

In some embodiments, after generating the tactile output in response todetecting the second user interaction, the device forgoes (1636)generation of a subsequent tactile output in response to detecting athird user interaction that is part of the same sequence of userinteractions as the first user interaction and the second userinteraction, in accordance with a determination that the third userinteraction occurred less than a predefined time duration from thesecond user interaction (e.g., as described in further detail hereinwith reference to FIGS. 5K-5O and 5JJJ, and method 1800).

In some embodiments, the device detects (1638) an end of the sequence ofuser interactions. In response to detecting the end of the sequence ofuser interactions, the device sets the one or more tactile outputgenerators to the low-power state.

In some embodiments, a monitoring function tracks (1640) a progress ofthe sequence of user interactions, and the one or more tactile outputgenerators are set to the low-power state based on a determination thatthe monitoring function has entered an end state (e.g., an end statefor, or with respect to, the sequence of user interactions).

It should be understood that the particular order in which theoperations in FIGS. 16A-16D have been described is merely an example andis not intended to indicate that the described order is the only orderin which the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein. Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,methods 600, 800, 1000, 1200, 1400, 1800, and 2000) are also applicablein an analogous manner to method 1600 described above with respect toFIGS. 16A-16D. For example, the tactile outputs, tactile outputgenerators, tactile output generator states, user interactions, userinterface elements, and user interface manipulations described abovewith reference to method 1600 optionally have one or more of thecharacteristics of the tactile outputs, tactile output generators,tactile output generator states, user interactions, user interfaceelements, and user interface manipulations described herein withreference to other methods described herein (e.g., methods 600, 800,1000, 1200, 1400, 1800, and 2000). For brevity, these details are notrepeated here.

The operations in the information processing methods described aboveare, optionally implemented by running one or more functional modules ininformation processing apparatus such as general purpose processors(e.g., as described above with respect to FIGS. 1A and 3) or applicationspecific chips.

In accordance with some embodiments, FIG. 17 shows a functional blockdiagram of electronic device 1700 configured in accordance with theprinciples of the various described embodiments. The functional blocksof the device are, optionally, implemented by hardware, software, or acombination of hardware and software to carry out the principles of thevarious described embodiments. It is understood by persons of skill inthe art that the functional blocks described in FIG. 17 are, optionally,combined or separated into sub-blocks to implement the principles of thevarious described embodiments. Therefore, the description hereinoptionally supports any possible combination or separation or furtherdefinition of the functional blocks described herein.

As shown in FIG. 17, electronic device 1700 includes display unit 1702(e.g., including display 112) configured to display user interfaces,touch-sensitive surface unit 1704 configured to receive touch inputs(e.g., on a surface, such as a display surface of display unit 1702),one or more tactile output generator unit(s) 1706 configured to generateone or more tactile outputs, and processing unit 1708 coupled withdisplay unit 1702, touch-sensitive surface unit 1704, and one or moretactile output generator unit(s) 1706. In some embodiments, processingunit 1708 includes one or more of the following sub-units: displayenabling unit 1710, detecting unit 1712, state setting unit 1714,tactile output enabling unit 1716, determining unit 1718, and scrollenabling unit 1720.

In some embodiments, processing unit 1708 is configured to enabledisplay (e.g., using display enabling unit 1710) of a user interface ondisplay unit 1702. While enabling display of the user interface ondisplay unit 1702 and while one or more tactile output generator units1706 are in a low-power state, processing unit 1708 detects (e.g., usingdetecting unit 1712) a first user interaction via touch-sensitivesurface unit 1704. Processing unit 1708 is configured to, in response todetecting the first user interaction, set one or more tactile outputgenerator units 1706 to a low-latency state (e.g., using state settingunit 1714). Processing unit 1708 is configured to, after setting one ormore tactile output generator units 1706 to the low-latency state,detect (e.g., using detecting unit 1712) a second user interaction thatis part of a same sequence of user interactions as the first userinteraction. Processing unit 1708 is configured to, in response todetecting the second user interaction, enable generation of a tactileoutput (e.g., using tactile output enabling unit 1716) that correspondsto the second user interaction.

In some embodiments, the first user interaction and the second userinteraction are detected as part of a continuous touch input ontouch-sensitive surface unit 1704 that traverses a sequence ofcomponents of the user interface displayed on display unit 1702.

In some embodiments, the first user interaction is touchdown of acontact on touch-sensitive surface unit 1704. In some embodiments, thefirst user interaction is a change in a contact after detectingtouchdown of the contact on touch-sensitive surface unit 1704.

In some embodiments, a monitoring function tracks a progress of thesequence of user interactions. One or more tactile output generatorunits 1706 are changed (e.g., using state setting unit 1714) to thelow-latency state based on a determination (e.g., made using determiningunit 1718) that the monitoring function has entered a start state inwhich the monitoring function has started monitoring the sequence ofuser interactions, and the tactile output is generated (e.g., usingtactile output enabling unit 1716) based on a determination (e.g., madeusing determining unit 1718) that the monitoring function has entered achanged state.

In some embodiments, a monitoring function tracks a progress of thesequence of user interactions. One or more tactile output generatorunits 1706 are changed (e.g., using state setting unit 1714) to thelow-latency state based on a determination (e.g., made using determiningunit 1718) that the monitoring function has entered a changed state thatprecedes a changed state that corresponds to a tactile output, and thetactile output is generated (e.g., using tactile output enabling unit1716) based on a determination (e.g., made using determining unit 1718)that the monitoring function has entered the changed state thatcorresponds to a tactile output.

In some embodiments, processing unit 1708 is further configured todetect (e.g., using detecting unit 1712) an end of the sequence of userinteractions. Processing unit 1708 is configured to, in response todetecting the end of the sequence of user interactions, set one or moretactile output generator units 1706 to the low-power state (e.g., usingstate setting unit 1714).

In some embodiments, a monitoring function tracks a progress of thesequence of user interactions, and one or more tactile output generatorunits 1706 are set (e.g., using state setting unit 1714) to thelow-power state based on a determination (e.g., made using determiningunit 1718) that the monitoring function has entered an end state.

In some embodiments, the sequence of user interactions corresponds to acontinuous touch input on touch-sensitive surface unit 1704 thattraverses a sequence of components of a user interface displayed ondisplay unit 1702, and processing unit 1708 is further configured toenable generation of a sequence of two or more tactile outputs (e.g.,using tactile output enabling unit 1716) using identical waveforms.

In some embodiments, processing unit 1708 is configured to, afterenabling generation of the tactile output in response to detecting thesecond user interaction, forgo generation of a subsequent tactile output(e.g., using tactile output enabling unit 1716) in response to detecting(e.g., using detecting unit 1712) a third user interaction that is partof the same sequence of user interactions as the first user interactionand the second user interaction, in accordance with a determination(e.g., made using determining unit 1718) that the third user interactionoccurred less than a predefined time duration from the second userinteraction.

In some embodiments, processing unit 1708 enables scrolling of a set ofmenu items (e.g., using scroll enabling unit 1720) in response to userinput on touch-sensitive surface unit 1704, wherein the sequence of userinteractions comprises a sequence of respective menu items in the set ofmenu items scrolling to or through a selection position in the userinterface.

In some embodiments, processing unit 1708 enables scrolling a set ofdate component or time component items (e.g., using scroll enabling unit1720) in response to user input on touch-sensitive surface unit 1704,wherein the sequence of user interactions comprises a sequence of datecomponent or time component items scrolling to or through a selectionposition in the user interface.

In some embodiments, the sequence of user interactions corresponds to atouch input on touch-sensitive surface unit 1704 that traverses asequence of soft keys displayed on display unit 1702, and processingunit 1708 enables generation of one or more tactile outputs (e.g., usingtactile output enabling unit 1716) that correspond to the touch inputreaching predefined positions with respect to the sequence of soft keys.

In some embodiments, the sequence of user interactions includes asequence of events corresponding to distinct phases of dragging anobject, in the user interface displayed on display unit 1702, from aninitial location to another location. In some embodiments, the distinctphases of dragging the object include an object lifting phase and anobject dropping phase. In some embodiments, the distinct phases ofdragging the object include an object snapping phase, corresponding tothe object snapping into place with respect to a user interface feature,and the tactile output corresponds to the object snapping into placewith respect to the user interface feature.

The operations described above with reference to FIGS. 16A-16D are,optionally, implemented by components depicted in FIGS. 1A-1B or FIG.17. For example, detecting operation 1604, setting operation 1610, andgenerating operation 1614 are, optionally, implemented by event sorter170, event recognizer 180, and event handler 190. Event monitor 171 inevent sorter 170 detects a contact on touch-sensitive display 112, andevent dispatcher module 174 delivers the event information toapplication 136-1. A respective event recognizer 180 of application136-1 compares the event information to respective event definitions186, and determines whether a first contact at a first location on thetouch-sensitive surface (or whether rotation of the device) correspondsto a predefined event or sub-event, such as selection of an object on auser interface, or rotation of the device from one orientation toanother. When a respective predefined event or sub-event is detected,event recognizer 180 activates an event handler 190 associated with thedetection of the event or sub-event. Event handler 190 optionally usesor calls data updater 176 or object updater 177 to update theapplication internal state 192. In some embodiments, event handler 190accesses a respective GUI updater 178 to update what is displayed by theapplication. Similarly, it would be clear to a person having ordinaryskill in the art how other processes can be implemented based on thecomponents depicted in FIGS. 1A-1B.

FIGS. 18A-18C are flow diagrams illustrating method 1800 ofconditionally generating tactile outputs based on a time interval from aprevious tactile output generation in accordance with some embodiments.Method 1800 is performed at an electronic device (e.g., device 300, FIG.3, or portable multifunction device 100, FIG. 1A) with a display, atouch-sensitive surface, and one or more sensors to detect intensity ofcontacts with the touch-sensitive surface. In some embodiments, thedisplay is a touch-screen display and the touch-sensitive surface is onor integrated with the display. In some embodiments, the display isseparate from the touch-sensitive surface. Some operations in method1800 are, optionally, combined and/or the order of some operations is,optionally, changed.

As described below, method 1800 provides an enhanced way to providetactile outputs based on time intervals. Forgoing generation of tactileoutputs based on time intervals from preceding tactile outputs reducesexcessive generation of tactile outputs, thereby protecting tactileoutput generators. In addition, forgoing generation of tactile outputsbased on time intervals from preceding tactile outputs reducesoverloading the user with tactile outputs, thereby allowing the user tofocus on more important tactile outputs. Thus, this protects the deviceand makes the user-device interface more efficient (e.g., by providingmore important tactile outputs and reducing user mistakes and/orunintended operations when operating/interacting with the device).

The device receives (1802) a first request for a first user interfaceoperation.

The first user interface operation is (1804) a first type of change in arespective user interface element that is associated with a firsttactile output.

In response to receiving the first request (1806), the device performs(1808) the first user interface operation, and generates (1810) thefirst tactile output.

After performing the first user interface operation, the device receives(1812) a second request for a second user interface operation that isassociated with a second tactile output.

In response to receiving the second request (1814), in accordance with adetermination (1816) that the second request for the second userinterface operation comprises a request for an instance of the firsttype of change in the user interface that is associated with the secondtactile output, the operations described as followed are performed(e.g., as described in further detail herein with reference to FIG.5JJJ).

The device determines (1818) a time interval from a point in timecorresponding to a most recent prior instance of the first type ofchange in the respective user interface element for which a tactileoutput was generated to a point in time corresponding to the first typeof change in the respective user interface element requested by thesecond request. In some embodiments, the time interval is measured fromrequest times for the tactile outputs, times associated with userinterface changes, or tactile output times (actual and projected).

In accordance with a determination (1820) that the time interval is lessthan a predefined time period, the device performs the second userinterface operation without generating the second tactile output (e.g.,as described in further detail herein with reference to the requests andtactile outputs in tactile output graph 5170 a, FIG. 5JJJ).

In some embodiments, the time interval is (1822) determined from abeginning of the tactile output generated for the most recent priorinstance of the first type of change in the respective user interfaceelement to a beginning of the second tactile output. In someembodiments, the predefined time period is longer than a duration of thetactile output generated for the most recent prior instance of the firsttype of change in the respective user interface element.

In accordance with a determination (1824) that the time interval isgreater than the predefined time period, the device performs the seconduser interface operation and generates the second tactile output (e.g.,as described in further detail herein with reference to the requests andtactile outputs in tactile output graph 5170 a, FIG. 5JJJ).

In some embodiments, in accordance (1826) a determination that thesecond request for the second user interface operation does not comprisea request for an instance of the first type of change in the respectiveuser interface element, the device mixes the second tactile output withother tactile outputs generated by the one or more tactile outputgenerators without regard to whether the time interval is less than thepredefined time period (e.g., as described in further detail herein withreference to the requests and tactile outputs in tactile output graph5170 b, FIG. 5JJJ).

In some embodiments, after performing the second user interfaceoperation, the device receives (1828) a third request for a third userinterface operation that is associated with a third tactile output. Inresponse to receiving the third request for the third user interfaceoperation, in accordance with a determination that the third request forthe third user interface operation comprises a request for an instanceof the first type of change in the user interface that is associatedwith a third tactile output, the device determines a second timeinterval from a point in time corresponding to a most recent priorinstance of the first type of change in the respective user interfaceelement for which a tactile output was generated to a point in timecorresponding to the first type of change in the respective userinterface element requested by the third request. In accordance with adetermination that the second time interval is less than the predefinedtime period, the device performs the third user interface operationwithout generating the third tactile output. In accordance with adetermination that the second time interval is greater than thepredefined time period, the device performs the third user interfaceoperation and generates the third tactile output.

In some embodiments, the first and third tactile outputs are (1830)synchronized with first and third user interactions in a sequence ofuser interactions that correspond to a continuous touch input on thetouch-sensitive surface (e.g., traversing a sequence of user interfaceelements, such as a sequence of menu items, date or time componentitems, or soft keys, in response to a continuous touch input). In someembodiments, the first, second, and third tactile outputs correspond tofirst, second, and third elements, respectively, in the sequence. Insome embodiments, generation of a respective tactile output issynchronized with the continuous touch input traversing thecorresponding respective user interface element. In some embodiments,however, if the input traverses the sequence too quickly, one or more ofthe tactile outputs are not generated. In some embodiments, even if, forexample, the second tactile output is not generated, generation of thefirst and third tactile outputs is synchronized with the continuoustouch input traversing the first and third element, respectively (e.g.,without regard to whether the second tactile output is generated).

In some embodiments, in response to receiving the third request (1832),in accordance with a determination that the third request for the thirduser interface operation does not comprise an instance of the first typeof change in the respective user interface element, the device mixes thethird tactile output with other tactile outputs generated by the one ormore tactile output generators without regard to whether the second timeinterval is less than the predefined time period (e.g., as described infurther detail herein with reference to the requests and tactile outputsin tactile output graphs 517 a and 5170 b, FIG. 5JJJ).

It should be understood that the particular order in which theoperations in FIGS. 18A-18C have been described is merely an example andis not intended to indicate that the described order is the only orderin which the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein. Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,methods 600, 800, 1000, 1200, 1400, 1600, and 2000) are also applicablein an analogous manner to method 1800 described above with respect toFIGS. 18A-18C. For example, the user interface elements, user interfaceoperations, and tactile outputs, described above with reference tomethod 1800 optionally have one or more of the characteristics of theuser interface elements, user interface operations, and tactile outputsdescribed herein with reference to other methods described herein (e.g.,methods 600, 800, 1000, 1200, 1400, 1600, and 2000). For brevity, thesedetails are not repeated here.

The operations in the information processing methods described aboveare, optionally implemented by running one or more functional modules ininformation processing apparatus such as general purpose processors(e.g., as described above with respect to FIGS. 1A and 3) or applicationspecific chips.

In accordance with some embodiments, FIG. 19 shows a functional blockdiagram of electronic device 1900 configured in accordance with theprinciples of the various described embodiments. The functional blocksof the device are, optionally, implemented by hardware, software, or acombination of hardware and software to carry out the principles of thevarious described embodiments. It is understood by persons of skill inthe art that the functional blocks described in FIG. 19 are, optionally,combined or separated into sub-blocks to implement the principles of thevarious described embodiments. Therefore, the description hereinoptionally supports any possible combination or separation or furtherdefinition of the functional blocks described herein.

As shown in FIG. 19, electronic device 1900 includes display unit 1902(e.g., including display 112) configured to display user interfaces,touch-sensitive surface unit 1904 configured to receive touch inputs(e.g., on a surface, such as a display surface of display unit 1902),one or more tactile output generator unit(s) 1906 configured to generateone or more tactile outputs, and processing unit 1908 coupled withdisplay unit 1902, touch-sensitive surface unit 1904, and one or moretactile output generator unit(s) 1906. In some embodiments, processingunit 1908 includes one or more of the following sub-units: receivingunit 1910, performing unit 1912, tactile output enabling unit 1914,determining unit 1916, mixing unit 1918, and synchronizing unit 1920.

In some embodiments, processing unit 1908 is configured to receive(e.g., using receiving unit 1910) a first request for a first userinterface operation, wherein the first user interface operation is afirst type of change in a respective user interface element that isassociated with a first tactile output. Processing unit 1908 isconfigured to, in response to receiving the first request, perform(e.g., using performing unit 1912) the first user interface operation,and enable generation of the first tactile output (e.g., using tactileoutput enabling unit 1914). Processing unit 1908 is configured to, afterperforming the first user interface operation, receive (e.g., usingreceiving unit 1910) a second request for a second user interfaceoperation that is associated with a second tactile output. Processingunit 1908 is configured to, in response to receiving the second request,in accordance with a determination (e.g., made using determining unit1916) that the second request for the second user interface operationcomprises a request for an instance of the first type of change in theuser interface that is associated with a second tactile output,determine a time interval (e.g., using determining unit 1916) from apoint in time corresponding to a most recent prior instance of the firsttype of change in the respective user interface element for which atactile output was generated to a point in time corresponding to thefirst type of change in the respective user interface element requestedby the second request. Processing unit 1908 is configured to, inaccordance with a determination (e.g., made using determining unit 1916)that the time interval is less than a predefined time period, performthe second user interface operation (e.g., using performing unit 1912)without enabling generation of the second tactile output; and, inaccordance with a determination that the time interval is greater thanthe predefined time period, perform the second user interface operation(e.g., using performing unit 1912) and enable generation of the secondtactile output (e.g., using tactile output enabling unit 1914).

In some embodiments, the time interval is determined (e.g., usingdetermining unit 1916) from a beginning of the tactile output generatedfor the most recent prior instance of the first type of change in therespective user interface element to a beginning of the second tactileoutput, and the predefined time period is longer than a duration of thetactile output generated for the most recent prior instance of the firsttype of change in the respective user interface element.

In some embodiments, processing unit 1908 is configured to, in responseto receiving the second request, in accordance with a determination(e.g., made using determining unit 1916) that the second request for thesecond user interface operation does not comprise a request for aninstance of the first type of change in the respective user interfaceelement, mix (e.g., using mixing unit 1918 and/or tactile outputenabling unit 1914) the second tactile output with other tactile outputsgenerated by one or more tactile output generator units 1906 withoutregard to whether the time interval is less than the predefined timeperiod.

In some embodiments, processing unit 1908 is configured to, afterperforming the second user interface operation, receive (e.g., usingreceiving unit 1910) a third request for third user interface operationthat is associated with a third tactile output. Processing unit 1908 isconfigured to, in response to receiving the third request for a thirduser interface operation, in accordance with a determination (e.g., madeusing determining unit 1916) that the third request for the third userinterface operation comprises a request for an instance of the firsttype of change in the user interface that is associated with a thirdtactile output, determine (e.g., using determining unit 1916) a secondtime interval from a point in time corresponding to a most recent priorinstance of the first type of change in the respective user interfaceelement for which a tactile output was generated to a point in timecorresponding to the first type of change in the respective userinterface element requested by the third request. Processing unit 1908is configured to, in accordance with a determination (e.g., made usingdetermining unit 1916) that the second time interval is less than thepredefined time period, perform the third user interface operation(e.g., using performing unit 1912) without enabling generation of thethird tactile output. Processing unit 1908 is configured to, inaccordance with a determination (e.g., made using determining unit 1916)that the second time interval is greater than the predefined timeperiod, perform the third user interface operation (e.g., usingperforming unit 1912) and enable generation of the third tactile output(e.g., using tactile output enabling unit 1914).

In some embodiments, processing unit 1908 is configured to, in responseto receiving the third request, in accordance with a determination thatthe third request for the third user interface operation does notcomprise an instance of the first type of change in the respective userinterface element, mix (e.g., using mixing unit 1918 and/or tactileoutput enabling unit 1914) the third tactile output with other tactileoutputs generated by one or more tactile output generator units 1906without regard to whether the second time interval is less than thepredefined time period.

In some embodiments, the first and third tactile outputs aresynchronized (e.g., using synchronizing unit 1920, mixing unit 1918,and/or tactile output enabling unit 1914) with first and third userinteractions in a sequence of user interactions that correspond to acontinuous touch input on touch-sensitive surface unit 1904.

The operations described above with reference to FIGS. 18A-18C are,optionally, implemented by components depicted in FIGS. 1A-1B or FIG.19. For example, receiving operation 1802, performing operation 1808,and generating operation 1810 are, optionally, implemented by eventsorter 170, event recognizer 180, and event handler 190. Event monitor171 in event sorter 170 detects a contact on touch-sensitive display112, and event dispatcher module 174 delivers the event information toapplication 136-1. A respective event recognizer 180 of application136-1 compares the event information to respective event definitions186, and determines whether a first contact at a first location on thetouch-sensitive surface (or whether rotation of the device) correspondsto a predefined event or sub-event, such as selection of an object on auser interface, or rotation of the device from one orientation toanother. When a respective predefined event or sub-event is detected,event recognizer 180 activates an event handler 190 associated with thedetection of the event or sub-event. Event handler 190 optionally usesor calls data updater 176 or object updater 177 to update theapplication internal state 192. In some embodiments, event handler 190accesses a respective GUI updater 178 to update what is displayed by theapplication. Similarly, it would be clear to a person having ordinaryskill in the art how other processes can be implemented based on thecomponents depicted in FIGS. 1A-1B.

FIGS. 20A-20B are flow diagrams illustrating method 2000 of generatingtactile outputs based on a magnitude of an input and dimensions of auser interface element in accordance with some embodiments. Method 2000is performed at an electronic device (e.g., device 300, FIG. 3, orportable multifunction device 100, FIG. 1A) with a display, atouch-sensitive surface, and one or more sensors to detect intensity ofcontacts with the touch-sensitive surface. In some embodiments, thedisplay is a touch-screen display and the touch-sensitive surface is onor integrated with the display. In some embodiments, the display isseparate from the touch-sensitive surface. Some operations in method2000 are, optionally, combined and/or the order of some operations is,optionally, changed.

As described below, method 2000 provides an enhanced way to providetactile outputs using an application-independent software module.Providing tactile outputs using application-independent software basedon information from application software provides common user interfaceframework that provides consistent user experience when various softwareapplications are used. Providing a common user interface frame work tothe user enhances the usability of such software applications and thedevice executing such software applications. In addition, method 2000provides an intuitive way to provide tactile outputs based on amagnitude of an input. Tactile outputs based on the magnitude of theinput conform to user expectations for tactile outputs. In turn, thesefeatures enhance the operability of the device and make the user-deviceinterface more efficient (e.g., by helping the user to provide properinputs and reducing user mistakes and/or unintended operations whenoperating/interacting with the device). In addition, the method reducesthe size of software applications and makes execution of such softwareapplications faster. For battery-operated electronic devices, enabling auser to use software applications faster and more efficiently conservespower and increases the time between battery charges.

The device receives (2002), at an application-independent module (e.g.,application-independent software module 260, FIG. 1D), user interfaceinformation (e.g., information 272, FIG. 1D) from an application.

In some embodiments, the application-independent module comprises(2004), or is included in, an operating system of the electronic device(e.g., operating system 126, FIGS. 1A and 3).

The user interface information corresponds (2006) to one or moredisplayed user interface elements with one or more dimensions defined byan application-specific module of the application.

In some embodiments, the one or more dimensions of the user interfaceelements include (2008) an extent of a user interface element and anextent that the user interface element can be temporarily scrolledbeyond an edge of a display region (e.g., an edge of the display), asdescribed in further detail herein with reference to FIGS. 5P-5Y. Forexample, the scrolling beyond the edge of the display region ismaintained until the end of the input that caused the scrolling beyondthe edge of the display region or is based on simulated momentum of theuser interface element in response to an input with velocity in thedirection of scrolling, at which point the user interface elementscrolls back so that the user interface element is completely within thedisplay region. In some embodiments, a tactile output is generated whenthe user interface element reaches the minimum zoom size. In someembodiments, a tactile output is generated when the user interfaceelement reaches the amount by which the user interface element can betemporarily zoomed beyond the minimum zoom size. In some embodiments, atactile output is generated when the user interface element snaps backto the minimum zoom size after being zoomed beyond the minimum zoomsize.

In some embodiments, the one or more dimensions of the user interfaceelements include (2010) a maximum zoom size of a user interface elementand an amount by which the user interface element can be temporarilyzoomed beyond the maximum zoom size (e.g., the zooming beyond themaximum zoom size is maintained until the end of the input that causedthe zooming beyond the maximum zoom size, at which point the userinterface element returns to the maximum zoom size). In someembodiments, a tactile output is generated when the user interfaceelement reaches the maximum zoom size, as described in further detailherein with reference to FIGS. 5Z-5EE. In some embodiments, a tactileoutput is generated when the user interface element reaches the amountby which the user interface element can be temporarily zoomed beyond themaximum zoom size. In some embodiments, a tactile output is generatedwhen the user interface element snaps back to the maximum zoom sizeafter being zoomed beyond the maximum zoom size.

In some embodiments, the one or more dimensions of the user interfaceelements include (2012) a minimum zoom size of a user interface elementand an amount by which the user interface element can be temporarilyzoomed beyond the minimum zoom size (e.g., the zooming beyond theminimum zoom size is maintained until the end of the input that causedthe zooming beyond the minimum zoom size, at which point the userinterface element returns to the minimum zoom size). In someembodiments, a tactile output is generated when the user interfaceelement reaches the minimum zoom size, as described in further detailherein with reference to FIGS. 5F-5I. In some embodiments, a tactileoutput is generated when the user interface element reaches the amountby which the user interface element can be temporarily zoomed beyond theminimum zoom size. In some embodiments, a tactile output is generatedwhen the user interface element snaps back to the minimum zoom sizeafter being zoomed beyond the minimum zoom size.

Next, the device receives (2014) an input directed toward one or more ofthe displayed user interface elements (e.g., corresponding to changes ina position of a user input). In some embodiments, the device detectschanges in the position of the input by receiving them from anapplication or from other channel(s) (e.g., a module that is updatingthe UI for the application).

Next, the device determines (2016), at the application-independentmodule, one or more tactile outputs to be generated based on a magnitude(e.g., distance, velocity, intensity, etc.) of the input and the one ormore dimensions defined by the applications-specific module. In someembodiments, the device determines the one or more tactile outputs inaccordance with changes to a position or scale of one or more of thedisplayed user interface elements, based on a magnitude (or moregenerally a parameter) of the input.

Next, the device generates (2018), using the one or more tactile outputgenerators, the determined one or more tactile outputs (e.g., using aUIEdgeFeedbackGenerator class described in Appendix B or its instance).

In some embodiments, the application-independent module determines(2020) a magnitude of the one or more tactile outputs by applying apredefined physics model to one or more parameters of the received input(e.g., a magnitude of the velocity of the input).

In some embodiments, the application-independent module determines(2022) whether to generate a respective tactile output based on whetheran edge of a respective element, of the displayed user interfaceelements, has passed a predefined boundary (e.g., as described infurther detail herein with reference to FIGS. 5P-5II and 5NN-5PP).

In some embodiments, the application-independent module determines(2024) whether to generate a respective tactile output based on whethera parameter (e.g., size) of a respective element, of the displayed userinterface elements, has passed a predefined limit.

In some embodiments, the application-independent module determines(2026) whether to generate a respective tactile output based on whetheran edge of a respective element, of the displayed user interfaceelements, has passed a predefined boundary, and whether a respectiveuser touch input is detected (e.g., scrolling with a rubber band effectwhile a contact is detected, or performing a zoom in or zoom outoperation while a contact is detected).

It should be understood that the particular order in which theoperations in FIGS. 20A-20B have been described is merely an example andis not intended to indicate that the described order is the only orderin which the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein. Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,methods 600, 800, 1000, 1200, 1400, 1600, and 1800) are also applicablein an analogous manner to method 2000 described above with respect toFIGS. 20A-20B. For example, the tactile outputs, user interfaceelements, user interface operations, user interface information,software applications, application-independent modules, magnitudes oftactile outputs described above with reference to method 2000 optionallyhave one or more of the characteristics of the tactile outputs, userinterface elements, user interface operations, user interfaceinformation, software applications, application-independent modules,magnitudes of tactile outputs described herein with reference to othermethods described herein (e.g., methods 600, 800, 1000, 1200, 1400,1600, and 1800). For brevity, these details are not repeated here.

The operations in the information processing methods described aboveare, optionally implemented by running one or more functional modules ininformation processing apparatus such as general purpose processors(e.g., as described above with respect to FIGS. 1A and 3) or applicationspecific chips.

In accordance with some embodiments, FIG. 21 shows a functional blockdiagram of an electronic device 2100 configured in accordance with theprinciples of the various described embodiments. The functional blocksof the device are, optionally, implemented by hardware, software, or acombination of hardware and software to carry out the principles of thevarious described embodiments. It is understood by persons of skill inthe art that the functional blocks described in FIG. 21 are, optionally,combined or separated into sub-blocks to implement the principles of thevarious described embodiments. Therefore, the description hereinoptionally supports any possible combination or separation or furtherdefinition of the functional blocks described herein.

As shown in FIG. 21, an electronic device 2100 includes display unit2102 (e.g., corresponding to display 112) configured to display userinterfaces, touch-sensitive surface unit 2104 configured to receivetouch inputs (e.g., on a surface, such as a display surface of displayunit 2102), one or more tactile output generator unit(s) 2106 configuredto generate one or more tactile outputs, and processing unit 2108coupled with display unit 2102, the touch-sensitive surface unit 2104,and the tactile output generator unit(s) 2106. In some embodiments,processing unit 2108 includes one or more of the following sub-units:information receiving unit 2110, input receiving unit 211, determiningunit 2114, and tactile output enabling unit 2116.

In some embodiments, processing unit 2108 is configured to receive, atan application-independent module, user interface information from anapplication (e.g., using information receiving unit 2110), wherein theuser interface information corresponds to one or more displayed userinterface elements with one or more dimensions defined by anapplication-specific module of the application. Processing unit 2108 isconfigured to receive (e.g., using input receiving unit 2112) an inputdirected toward one or more of the displayed user interface elements.Processing unit 2108 is configured to determine, at theapplication-independent module, one or more tactile outputs to begenerated based on a magnitude of the input and the one or moredimensions defined by the applications-specific module (e.g., usingdetermining unit 2114), and enable generation (e.g., using tactileoutput enabling unit 2116) of the determined one or more tactile outputsusing one or more tactile output generator units 2106.

In some embodiments, the application-independent module comprises, or isincluded in, an operating system of electronic device 2100. In someembodiments, the application-independent module determines a magnitudeof the one or more tactile outputs by applying a predefined physicsmodel to one or more parameters of the received input.

In some embodiments, the application-independent module determines(e.g., using determining unit 2114) whether to generate a respectivetactile output based on whether an edge of a respective element, of thedisplayed user interface elements, has passed a predefined boundary. Insome embodiments, the application-independent module determines (e.g.,using determining unit 2114) whether to generate a respective tactileoutput based on whether a parameter of a respective element, of thedisplayed user interface elements, has passed a predefined limit. Insome embodiments, the application-independent module determines (e.g.,using determining unit 2114) whether to generate a respective tactileoutput based on whether an edge of a respective element, of thedisplayed user interface elements, has passed a predefined boundary, andwhether a respective user touch input is detected.

In some embodiments, the dimensions of the user interface elementsinclude an extent of a user interface element and an extent that theuser interface element can be temporarily scrolled beyond an edge of adisplay region. In some embodiments, the dimensions of the userinterface elements include a maximum zoom size of a user interfaceelement and an amount by which the user interface element can betemporarily zoomed beyond the maximum zoom size. In some embodiments,the dimensions of the user interface elements include a minimum zoomsize of a user interface element and an amount by which the userinterface element can be temporarily zoomed beyond the minimum zoomsize.

The operations described above with reference to FIGS. 20A-20B are,optionally, implemented by components depicted in FIGS. 1A-1B or FIG.21. For example, receiving operation 2002, determining operation 2016,and generating operation 2018 are, optionally, implemented by eventsorter 170, event recognizer 180, and event handler 190. Event monitor171 in event sorter 170 detects a contact on touch-sensitive display112, and event dispatcher module 174 delivers the event information toapplication 136-1. A respective event recognizer 180 of application136-1 compares the event information to respective event definitions186, and determines whether a first contact at a first location on thetouch-sensitive surface (or whether rotation of the device) correspondsto a predefined event or sub-event, such as selection of an object on auser interface, or rotation of the device from one orientation toanother. When a respective predefined event or sub-event is detected,event recognizer 180 activates an event handler 190 associated with thedetection of the event or sub-event. Event handler 190 optionally usesor calls data updater 176 or object updater 177 to update theapplication internal state 192. In some embodiments, event handler 190accesses a respective GUI updater 178 to update what is displayed by theapplication. Similarly, it would be clear to a person having ordinaryskill in the art how other processes can be implemented based on thecomponents depicted in FIGS. 1A-1B.

The foregoing description, for purpose of explanation, has beendescribed with reference to specific embodiments. However, theillustrative discussions above are not intended to be exhaustive or tolimit the invention to the precise forms disclosed. Many modificationsand variations are possible in view of the above teachings. Theembodiments were chosen and described in order to best explain theprinciples of the invention and its practical applications, to therebyenable others skilled in the art to best use the invention and variousdescribed embodiments with various modifications as are suited to theparticular use contemplated.

What is claimed is:
 1. A method, comprising: at an electronic devicewith a display, a touch-sensitive surface, and one or more tactileoutput generators: receiving a first request for a first user interfaceoperation, wherein the first user interface operation is a first type ofchange in a respective user interface element that is associated with afirst tactile output; in response to receiving the first request:performing the first user interface operation; and generating the firsttactile output; after performing the first user interface operation,receiving a second request for a second user interface operation that isassociated with a second tactile output; in response to receiving thesecond request: in accordance with a determination that the secondrequest for the second user interface operation comprises a request foran instance of the first type of change in the user interface that isassociated with the second tactile output: determining a time intervalfrom a point in time corresponding to a most recent prior instance ofthe first type of change in the respective user interface element forwhich a tactile output was generated to a point in time corresponding tothe first type of change in the respective user interface elementrequested by the second request; in accordance with a determination thatthe time interval is less than a predefined time period, performing thesecond user interface operation without generating the second tactileoutput; and in accordance with a determination that the time interval isgreater than the predefined time period, performing the second userinterface operation and generating the second tactile output; and inaccordance with a determination that the second request for the seconduser interface operation does not comprise a request for an instance ofthe first type of change in the respective user interface element,mixing the second tactile output with other tactile outputs generated bythe one or more tactile output generators without regard to whether thetime interval is less than the predefined time period.
 2. The method ofclaim 1, wherein the time interval is determined from a beginning of thetactile output generated for the most recent prior instance of the firsttype of change in the respective user interface element to a beginningof the second tactile output, and the predefined time period is longerthan a duration of the tactile output generated for the most recentprior instance of the first type of change in the respective userinterface element.
 3. The method of claim 1, including: after performingthe second user interface operation, receiving a third request for athird user interface operation that is associated with a third tactileoutput; in response to receiving the third request for the third userinterface operation: in accordance with a determination that the thirdrequest for the third user interface operation comprises a request foran instance of the first type of change in the user interface that isassociated with a third tactile output: determining a second timeinterval from a point in time corresponding to a most recent priorinstance of the first type of change in the respective user interfaceelement for which a tactile output was generated to a point in timecorresponding to the first type of change in the respective userinterface element requested by the third request; in accordance with adetermination that the second time interval is less than the predefinedtime period, performing the third user interface operation withoutgenerating the third tactile output; and in accordance with adetermination that the second time interval is greater than thepredefined time period, performing the third user interface operationand generating the third tactile output.
 4. The method of claim 3,including, in response to receiving the third request: in accordancewith a determination that the third request for the third user interfaceoperation does not comprise an instance of the first type of change inthe respective user interface element, mixing the third tactile outputwith other tactile outputs generated by the one or more tactile outputgenerators without regard to whether the second time interval is lessthan the predefined time period.
 5. The method of claim 3, wherein thefirst and third tactile outputs are synchronized with first and thirduser interactions in a sequence of user interactions that correspond toa continuous touch input on the touch-sensitive surface.
 6. Anelectronic device, comprising: a display; a touch-sensitive surface; oneor more tactile output generators; one or more processors; and memorystoring one or more programs for execution by the one or moreprocessors, the one or more programs including instructions for:receiving a first request for a first user interface operation, whereinthe first user interface operation is a first type of change in arespective user interface element that is associated with a firsttactile output; in response to receiving the first request: performingthe first user interface operation; and generating the first tactileoutput; after performing the first user interface operation, receiving asecond request for a second user interface operation that is associatedwith a second tactile output; and in response to receiving the secondrequest: in accordance with a determination that the second request forthe second user interface operation comprises a request for an instanceof the first type of change in the user interface that is associatedwith a second tactile output:  determining a time interval from a pointin time corresponding to a most recent prior instance of the first typeof change in the respective user interface element for which a tactileoutput was generated to a point in time corresponding to the first typeof change in the respective user interface element requested by thesecond request;  in accordance with a determination that the timeinterval is less than a predefined time period, performing the seconduser interface operation without generating the second tactile output;and  in accordance with a determination that the time interval isgreater than the predefined time period, performing the second userinterface operation and generating the second tactile output; and inaccordance with a determination that the second request for the seconduser interface operation does not comprise a request for an instance ofthe first type of change in the respective user interface element,mixing the second tactile output with other tactile outputs generated bythe one or more tactile output generators without regard to whether thetime interval is less than the predefined time period.
 7. The electronicdevice of claim 6, wherein the time interval is determined from abeginning of the tactile output generated for the most recent priorinstance of the first type of change in the respective user interfaceelement to a beginning of the second tactile output, and the predefinedtime period is longer than a duration of the tactile output generatedfor the most recent prior instance of the first type of change in therespective user interface element.
 8. The electronic device of claim 6,wherein the one or more programs include instructions for: afterperforming the second user interface operation, receiving a thirdrequest for a third user interface operation that is associated with athird tactile output; in response to receiving the third request for thethird user interface operation: in accordance with a determination thatthe third request for the third user interface operation comprises arequest for an instance of the first type of change in the userinterface that is associated with a third tactile output: determining asecond time interval from a point in time corresponding to a most recentprior instance of the first type of change in the respective userinterface element for which a tactile output was generated to a point intime corresponding to the first type of change in the respective userinterface element requested by the third request; in accordance with adetermination that the second time interval is less than the predefinedtime period, performing the third user interface operation withoutgenerating the third tactile output; and in accordance with adetermination that the second time interval is greater than thepredefined time period, performing the third user interface operationand generating the third tactile output.
 9. The electronic device ofclaim 8, wherein the one or more programs include instructions for, inresponse to receiving the third request: in accordance with adetermination that the third request for the third user interfaceoperation does not comprise an instance of the first type of change inthe respective user interface element, mixing the third tactile outputwith other tactile outputs generated by the one or more tactile outputgenerators without regard to whether the second time interval is lessthan the predefined time period.
 10. The electronic device of claim 8,wherein the first and third tactile outputs are synchronized with firstand third user interactions in a sequence of user interactions thatcorrespond to a continuous touch input on the touch-sensitive surface.11. A computer readable storage medium storing one or more programs, theone or more programs comprising instructions, which, when executed by anelectronic device with a display, a touch-sensitive surface, and one ormore tactile output generators, cause the electronic device to: receivea first request for a first user interface operation, wherein the firstuser interface operation is a first type of change in a respective userinterface element that is associated with a first tactile output; inresponse to receiving the first request: perform the first userinterface operation; and generate the first tactile output; afterperforming the first user interface operation, receive a second requestfor a second user interface operation that is associated with a secondtactile output; and in response to receiving the second request: inaccordance with a determination that the second request for the seconduser interface operation comprises a request for an instance of thefirst type of change in the user interface that is associated with asecond tactile output: determine a time interval from a point in timecorresponding to a most recent prior instance of the first type ofchange in the respective user interface element for which a tactileoutput was generated to a point in time corresponding to the first typeof change in the respective user interface element requested by thesecond request; in accordance with a determination that the timeinterval is less than a predefined time period, perform the second userinterface operation without generating the second tactile output; and inaccordance with a determination that the time interval is greater thanthe predefined time period, perform the second user interface operationand generating the second tactile output; and in accordance with adetermination that the second request for the second user interfaceoperation does not comprise a request for an instance of the first typeof change in the respective user interface element, mixing the secondtactile output with other tactile outputs generated by the one or moretactile output generators without regard to whether the time interval isless than the predefined time period.
 12. The computer readable storagemedium of claim 8, wherein the time interval is determined from abeginning of the tactile output generated for the most recent priorinstance of the first type of change in the respective user interfaceelement to a beginning of the second tactile output, and the predefinedtime period is longer than a duration of the tactile output generatedfor the most recent prior instance of the first type of change in therespective user interface element.